You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by GitBox <gi...@apache.org> on 2020/09/24 12:39:17 UTC
[GitHub] [madlib] orhankislal opened a new pull request #519: DL: Add a helper function to load custom top n accuracy functions
orhankislal opened a new pull request #519:
URL: https://github.com/apache/madlib/pull/519
JIRA: MADLIB-1452
This commit enables the top_n_accuracy metric. The current parser
cannot use top_n_accuracy(k=3) format because we don't want to
run eval for security reasons. Instead, we add a helper function
so that the user can easily create a custom top_n_accuracy
function.
<!--
Thanks for sending a pull request! Here are some tips for you:
1. Refer to this link for contribution guidelines https://cwiki.apache.org/confluence/display/MADLIB/Contribution+Guidelines
2. Please Provide the Module Name, a JIRA Number and a short description about your changes.
-->
- [ ] Add the module name, JIRA# to PR/commit and description.
- [ ] Add tests for the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta merged pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta merged pull request #519:
URL: https://github.com/apache/madlib/pull/519
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #519:
URL: https://github.com/apache/madlib/pull/519#discussion_r498650951
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_custom_function.py_in
##########
@@ -146,6 +146,43 @@ def update_builtin_metrics(builtin_metrics):
builtin_metrics.append('ce')
return builtin_metrics
+@MinWarning("error")
+def load_top_n_accuracy_function(schema_madlib, object_table, n, is_sparse, **kwargs):
+
+ object_table = quote_ident(object_table)
+ _assert(n > 0,
+ "{0}: For top n accuracy functions n has to be a positive integer.".format(module_name))
+ sparse_prefix = "sparse_" if is_sparse else ""
+
+ sql = """
+ CREATE OR REPLACE FUNCTION __madlib__{sparse_prefix}top_{n}_acc_pickled()
Review comment:
Yes, this function is available only to the superusers. Calling the `load_custom_function` will be exclusive to the superusers as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-704635441
is this ready for final acceptance @khannaekta ?
thx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #519:
URL: https://github.com/apache/madlib/pull/519#discussion_r498652349
##########
File path: src/ports/postgres/modules/deep_learning/test/madlib_keras_custom_function.sql_in
##########
@@ -31,107 +31,130 @@ m4_include(`SQLCommon.m4')
)
/* Test successful table creation where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'INTEGER', 'id column should be INTEGER type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'id';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'BYTEA', 'object column should be BYTEA type' )
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'object';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'name column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'name';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'description column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'description';
/* id should be 1 */
SELECT assert(id = 1, 'Wrong id written by load_custom_function')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Validate function object created */
SELECT assert(read_custom_function(object, 2, 3) = 5, 'Custom function should return sum of args.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Test custom function insertion where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
SELECT assert(name = 'sum_fn', 'Custom function sum_fn found in table.')
- FROM test_custom_function_table WHERE id = 1;
+ FROM __test_custom_function_table__ WHERE id = 1;
SELECT assert(name = 'sum_fn1', 'Custom function sum_fn1 found in table.')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test adding an existing function name should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
$TRAP$) = 1, 'Should error out for duplicate function name');
/* Test deletion by id where valid table exists */
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'id 2 should exist before deletion!')
- FROM test_custom_function_table WHERE id = 2;
-SELECT delete_custom_function('test_custom_function_table', 2);
+ FROM __test_custom_function_table__ WHERE id = 2;
+SELECT delete_custom_function('__test_custom_function_table__', 2);
SELECT assert(COUNT(id) = 0, 'id 2 should have been deleted!')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test deletion by name where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'function name sum_fn1 should exist before deletion!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
-SELECT delete_custom_function('test_custom_function_table', 'sum_fn1');
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
+SELECT delete_custom_function('__test_custom_function_table__', 'sum_fn1');
SELECT assert(COUNT(id) = 0, 'function name sum_fn1 should have been deleted!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
/* Test deleting an already deleted entry should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT delete_custom_function('test_custom_function_table', 2);
+ SELECT delete_custom_function('__test_custom_function_table__', 2);
$TRAP$) = 1, 'Should error out for trying to delete an entry that does not exist');
/* Test delete drops the table after deleting last entry*/
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-SELECT delete_custom_function('test_custom_function_table', 1);
-SELECT assert(COUNT(relname) = 0, 'Table test_custom_function_table should have been deleted.')
- FROM pg_class where relname='test_custom_function_table';
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+SELECT delete_custom_function('__test_custom_function_table__', 1);
+SELECT assert(COUNT(relname) = 0, 'Table __test_custom_function_table__ should have been deleted.')
+ FROM pg_class where relname='__test_custom_function_table__';
/* Test deletion where empty table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-DELETE FROM test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+DELETE FROM __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting function in an empty table should generate an exception.');
/* Test deletion where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting a non-existent table should raise exception.');
/* Test where invalid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-ALTER TABLE test_custom_function_table DROP COLUMN id;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 2)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+ALTER TABLE __test_custom_function_table__ DROP COLUMN id;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 2)$$) = 1,
'Deleting an invalid table should generate an exception.');
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
'Passing an invalid table to load_custom_function() should raise exception.');
/* Test input validation */
-DROP TABLE IF EXISTS test_custom_function_table;
+DROP TABLE IF EXISTS __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), NULL, NULL);
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), NULL, NULL);
$$) = 1, 'Name cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', NULL, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', NULL, 'sum_fn', NULL);
$$) = 1, 'Function object cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', 'invalid_obj'::bytea, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', 'invalid_obj'::bytea, 'sum_fn', NULL);
$$) = 1, 'Invalid custom function object');
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', NULL);
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', NULL);
SELECT assert(name IS NOT NULL AND description IS NULL, 'validate name is not NULL.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT delete_custom_function('test_custom_function_table', NULL);
+ SELECT delete_custom_function('__test_custom_function_table__', NULL);
$$) = 1, 'id/name cannot be NULL!');
+
+/* Test top n accuracy */
+
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 3);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7, True);
Review comment:
I guess we can add a boolean parameter to the `load_custom_function`to disable this info message if it is called from the top_n. I don't think it is a big problem but I'm happy to make the change as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
```
load_top_k_accuracy_function(
object table,
k
)
```
because this is the terminology that Keras uses. Also make user docs changes from `n` to `k`
(1) multi-model, top k default
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
```
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) did you test with madlib.madlib_keras_fit() ?
(5) user docs need to be update for fit() and fit_multiple() regarding this function and how to use it. I can work on that but want to record it here since should go as part of this PR before we merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705885794
> (0) can we please rename this to :
>
> ```
> load_top_k_accuracy_function(
> object table,
> k
> )
> ```......
Sure, will make those changes
> (2) multi-model, top k custom function
>
> ```
> INFO: Keras Custom Function: Created new custom function table custom_function_table.
> CONTEXT: PL/Python function "load_custom_function"
> SQL statement "
> SELECT madlib.load_custom_function('custom_function_table',
> madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
> 'top_3_accuracy',
> 'returns top_3_accuracy');
> "
> ......
> results OK but please remove verbose output
Agreed! I had the same comment in my initial review too. These verbose INFOs are coming from calling `load_custom_function()`, so we should probably suppress those.
> (4) did you test with madlib.madlib_keras_fit() ?
I tested it manually. But will add an end-to-end test for testing it with fit, fit_multiple and evaluate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705885794
> (0) can we please rename this to :
>
> ```
> load_top_k_accuracy_function(
> object table,
> k
> )
> ```......
Sure, will make those changes
> (2) multi-model, top k custom function
>
> ```
> INFO: Keras Custom Function: Created new custom function table custom_function_table.
> CONTEXT: PL/Python function "load_custom_function"
> SQL statement "
> SELECT madlib.load_custom_function('custom_function_table',
> madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
> 'top_3_accuracy',
> 'returns top_3_accuracy');
> "
> ......
> results OK but please remove verbose output
Agreed! I had the same comment in my initial review too. These verbose INFOs are coming from calling `load_custom_function()`, so we should probably suppress those.
> (4) did you test with madlib.madlib_keras_fit() ?
I tested it manually. But will add an end-to-end test for testing it with fit, fit_multiple and evaluate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705885794
> (0) can we please rename this to :
>
> ```
> load_top_k_accuracy_function(
> object table,
> k
> )
> ```......
Sure, will make those changes
> (2) multi-model, top k custom function
>
> ```
> INFO: Keras Custom Function: Created new custom function table custom_function_table.
> CONTEXT: PL/Python function "load_custom_function"
> SQL statement "
> SELECT madlib.load_custom_function('custom_function_table',
> madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
> 'top_3_accuracy',
> 'returns top_3_accuracy');
> "
> ......
> results OK but please remove verbose output
Agreed! I had the same comment in my initial review too. These verbose INFOs are coming from calling `load_custom_function()`, so we should probably suppress those.
> (4) did you test with madlib.madlib_keras_fit() ?
I tested it manually. But will add an end-to-end test for testing it with fit, fit_multiple and evaluate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
```
load_top_k_accuracy_function(
object table,
k
)
```
because this is the terminology that Keras uses. Also make user docs changes from `n` to `k`
(1) multi-model, top k default
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
```
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) single-model, run fit() with custom function
did you test with madlib.madlib_keras_fit() ?
(5) user docs need to be update for fit() and fit_multiple() regarding this function and how to use it. I can work on that but want to record it here since should go as part of this PR before we merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705885794
> (0) can we please rename this to :
>
> ```
> load_top_k_accuracy_function(
> object table,
> k
> )
> ```......
Sure, will make those changes
> (2) multi-model, top k custom function
>
> ```
> INFO: Keras Custom Function: Created new custom function table custom_function_table.
> CONTEXT: PL/Python function "load_custom_function"
> SQL statement "
> SELECT madlib.load_custom_function('custom_function_table',
> madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
> 'top_3_accuracy',
> 'returns top_3_accuracy');
> "
> ......
> results OK but please remove verbose output
Agreed! I had the same comment in my initial review too. These verbose INFOs are coming from calling `load_custom_function()`, so we should probably suppress those.
> (4) did you test with madlib.madlib_keras_fit() ?
I tested it manually. But will add an end-to-end test for testing it with fit, fit_multiple and evaluate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-712486086
Retesting items from above
(0)
```
SELECT madlib.load_top_k_accuracy_function(
'test_custom_function_table',
10
);
```
produces
```
SELECT id, name, description FROM test_custom_function_table1 ORDER BY id;
id | name | description
----+------------------+--------------------------
1 | top_10_accuracy | returns top_10_accuracy
```
OK
(2)
```
SELECT madlib.load_top_k_accuracy_function('custom_function_table', 3);
INFO: Keras Custom Function: Added function 'top_3_accuracy' to 'custom_function_table' table
CONTEXT: PL/Python function "load_top_k_accuracy_function"
load_top_k_accuracy_function
------------------------------
(1 row)
Time: 1644.004 ms
```
OK
(5)
commit below for user docs
(6)
```
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_k_accuracy_function('custom_function_table',
3);
SELECT madlib.load_top_k_accuracy_function('custom_function_table',
10);
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
produces
```
id | name | description
----+-----------------+-------------------------
1 | top_3_accuracy | returns top_3_accuracy
2 | top_10_accuracy | returns top_10_accuracy
```
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on a change in pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on a change in pull request #519:
URL: https://github.com/apache/madlib/pull/519#discussion_r498997257
##########
File path: src/ports/postgres/modules/deep_learning/test/madlib_keras_custom_function.sql_in
##########
@@ -31,107 +31,130 @@ m4_include(`SQLCommon.m4')
)
/* Test successful table creation where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'INTEGER', 'id column should be INTEGER type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'id';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'BYTEA', 'object column should be BYTEA type' )
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'object';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'name column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'name';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'description column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'description';
/* id should be 1 */
SELECT assert(id = 1, 'Wrong id written by load_custom_function')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Validate function object created */
SELECT assert(read_custom_function(object, 2, 3) = 5, 'Custom function should return sum of args.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Test custom function insertion where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
SELECT assert(name = 'sum_fn', 'Custom function sum_fn found in table.')
- FROM test_custom_function_table WHERE id = 1;
+ FROM __test_custom_function_table__ WHERE id = 1;
SELECT assert(name = 'sum_fn1', 'Custom function sum_fn1 found in table.')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test adding an existing function name should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
$TRAP$) = 1, 'Should error out for duplicate function name');
/* Test deletion by id where valid table exists */
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'id 2 should exist before deletion!')
- FROM test_custom_function_table WHERE id = 2;
-SELECT delete_custom_function('test_custom_function_table', 2);
+ FROM __test_custom_function_table__ WHERE id = 2;
+SELECT delete_custom_function('__test_custom_function_table__', 2);
SELECT assert(COUNT(id) = 0, 'id 2 should have been deleted!')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test deletion by name where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'function name sum_fn1 should exist before deletion!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
-SELECT delete_custom_function('test_custom_function_table', 'sum_fn1');
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
+SELECT delete_custom_function('__test_custom_function_table__', 'sum_fn1');
SELECT assert(COUNT(id) = 0, 'function name sum_fn1 should have been deleted!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
/* Test deleting an already deleted entry should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT delete_custom_function('test_custom_function_table', 2);
+ SELECT delete_custom_function('__test_custom_function_table__', 2);
$TRAP$) = 1, 'Should error out for trying to delete an entry that does not exist');
/* Test delete drops the table after deleting last entry*/
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-SELECT delete_custom_function('test_custom_function_table', 1);
-SELECT assert(COUNT(relname) = 0, 'Table test_custom_function_table should have been deleted.')
- FROM pg_class where relname='test_custom_function_table';
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+SELECT delete_custom_function('__test_custom_function_table__', 1);
+SELECT assert(COUNT(relname) = 0, 'Table __test_custom_function_table__ should have been deleted.')
+ FROM pg_class where relname='__test_custom_function_table__';
/* Test deletion where empty table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-DELETE FROM test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+DELETE FROM __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting function in an empty table should generate an exception.');
/* Test deletion where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting a non-existent table should raise exception.');
/* Test where invalid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-ALTER TABLE test_custom_function_table DROP COLUMN id;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 2)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+ALTER TABLE __test_custom_function_table__ DROP COLUMN id;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 2)$$) = 1,
'Deleting an invalid table should generate an exception.');
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
'Passing an invalid table to load_custom_function() should raise exception.');
/* Test input validation */
-DROP TABLE IF EXISTS test_custom_function_table;
+DROP TABLE IF EXISTS __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), NULL, NULL);
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), NULL, NULL);
$$) = 1, 'Name cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', NULL, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', NULL, 'sum_fn', NULL);
$$) = 1, 'Function object cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', 'invalid_obj'::bytea, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', 'invalid_obj'::bytea, 'sum_fn', NULL);
$$) = 1, 'Invalid custom function object');
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', NULL);
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', NULL);
SELECT assert(name IS NOT NULL AND description IS NULL, 'validate name is not NULL.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT delete_custom_function('test_custom_function_table', NULL);
+ SELECT delete_custom_function('__test_custom_function_table__', NULL);
$$) = 1, 'id/name cannot be NULL!');
+
+/* Test top n accuracy */
+
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 3);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7, True);
Review comment:
I don't have a strong opinion on the `INFO` message, so please do as think makes sense or is convenient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
-- (0) can we please rename this to :
--
-- load_top_k_accuracy_function(
-- object table,
-- k
-- )
--
-- because this is the terminology that Keras uses
-- (1) multi-model, top k default
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
-- OK
-- (2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
-- results OK but please remove verbose output
-- (3) multi-model, run fit() with custom function
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
-- OK
-- (4) single-model, run fit() with custom function
-- did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
-- (0) can we please rename this to :
--
-- load_top_k_accuracy_function(
-- object table,
-- k
-- )
--
-- because this is the terminology that Keras uses
-- (1) multi-model, top k default
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
-- OK
-- (2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
-- results OK but please remove verbose output
-- (3) multi-model, run fit() with custom function
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
-- OK
-- (4) single-model, run fit() with custom function
-- did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-712504771
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-704636244
@fmcquillan99 Yes, this PR is ready for final acceptance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-706265521
sounds good, thanks @khannaekta
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
```
load_top_k_accuracy_function(
object table,
k
)
```
because this is the terminology that Keras uses. Also make user docs changes from `n` to `k`
(1) multi-model, top k default
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
```
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) single-model, run fit() with custom function
did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
load_top_k_accuracy_function(
object table,
k
)
because this is the terminology that Keras uses. Also make user docs changes from `n` to `k`
(1) multi-model, top k default
```
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) single-model, run fit() with custom function
did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
load_top_k_accuracy_function(
object table,
k
)
because this is the terminology that Keras uses
(1) multi-model, top k default
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) single-model, run fit() with custom function
did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #519:
URL: https://github.com/apache/madlib/pull/519#issuecomment-705841751
(0) can we please rename this to :
load_top_k_accuracy_function(
object table,
k
)
because this is the terminology that Keras uses. Also make user docs changes from `n` to `k`
(1) multi-model, top k default
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
]
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_k_categorical_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE -- use gpus
);
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]-------------+---------------------------------------------
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table |
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:00:11.766483
end_training_time | 2020-10-08 21:00:36.933537
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
Time: 8.248 ms
madlib=# SELECT * FROM iris_multi_model_info;
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {23.0940079689026}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.981531918048859
training_metrics | {1}
training_loss | {0.981531918048859}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_k_categorical_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.4646620750427}
metrics_type | {top_k_categorical_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.49222993850708
training_metrics | {1}
training_loss | {0.49222993850708}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(2) multi-model, top k custom function
DROP TABLE IF EXISTS custom_function_table;
SELECT madlib.load_top_n_accuracy_function('custom_function_table',
3);
```
INFO: Keras Custom Function: Created new custom function table custom_function_table.
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
INFO: Keras Custom Function: Added function top_3_accuracy to custom_function_table table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('custom_function_table',
madlib.top_k_categorical_acc_pickled(3, 'top_3_accuracy'),
'top_3_accuracy',
'returns top_3_accuracy');
"
PL/Python function "load_custom_function"
load_top_n_accuracy_function
------------------------------
(1 row)
```
SELECT id, name, description FROM custom_function_table ORDER BY id;
```
id | name | description
----+----------------+------------------------
1 | top_3_accuracy | returns top_3_accuracy
(1 row)
```
results OK but please remove verbose output
(3) multi-model, run fit() with custom function
DROP TABLE IF EXISTS mst_table, mst_table_summary;
SELECT madlib.load_model_selection_table('model_arch_library', -- model architecture table
'mst_table', -- model selection table output
ARRAY[1,2], -- model ids from model architecture table
ARRAY[ -- compile params
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']$$,
$$loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy']$$
],
ARRAY[ -- fit params
$$batch_size=4,epochs=1$$,
$$batch_size=8,epochs=1$$
],
'custom_function_table' -- custom table
);
SELECT * FROM mst_table ORDER BY mst_key;
```
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['top_3_accuracy'] | batch_size=8,epochs=1
(12 rows)
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
2, -- num_iterations
FALSE, -- use gpus
NULL,
NULL
);
SELECT * FROM iris_multi_model_summary;
```
source_table | iris_train_packed
validation_table |
model | iris_multi_model
model_info | iris_multi_model_info
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_selection_table | mst_table
object_table | custom_function_table
num_iterations | 2
metrics_compute_frequency | 2
warm_start | f
name |
description |
start_training_time | 2020-10-08 21:22:27.219617
end_training_time | 2020-10-08 21:22:52.310825
madlib_version | 1.18.0-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_iters | {2}
```
SELECT * FROM iris_multi_model_info;
```
-[ RECORD 1 ]------------+--------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {22.8225800991058}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.488488465547562
training_metrics | {1}
training_loss | {0.488488465547562}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
-[ RECORD 2 ]------------+--------------------------------------------------------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['top_3_accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {23.2352600097656}
metrics_type | {top_3_accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 1
training_loss_final | 0.713700234889984
training_metrics | {1}
training_loss | {0.713700234889984}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
etc
```
OK
(4) single-model, run fit() with custom function
did you test with madlib.madlib_keras_fit() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on a change in pull request #519: DL: Add a helper function to load custom top n accuracy functions
Posted by GitBox <gi...@apache.org>.
khannaekta commented on a change in pull request #519:
URL: https://github.com/apache/madlib/pull/519#discussion_r498537268
##########
File path: src/ports/postgres/modules/deep_learning/test/madlib_keras_custom_function.sql_in
##########
@@ -31,107 +31,130 @@ m4_include(`SQLCommon.m4')
)
/* Test successful table creation where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'INTEGER', 'id column should be INTEGER type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'id';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'BYTEA', 'object column should be BYTEA type' )
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'object';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'name column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'name';
SELECT assert(UPPER(atttypid::regtype::TEXT) = 'TEXT',
'description column should be TEXT type')
- FROM pg_attribute WHERE attrelid = 'test_custom_function_table'::regclass
+ FROM pg_attribute WHERE attrelid = '__test_custom_function_table__'::regclass
AND attname = 'description';
/* id should be 1 */
SELECT assert(id = 1, 'Wrong id written by load_custom_function')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Validate function object created */
SELECT assert(read_custom_function(object, 2, 3) = 5, 'Custom function should return sum of args.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
/* Test custom function insertion where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
SELECT assert(name = 'sum_fn', 'Custom function sum_fn found in table.')
- FROM test_custom_function_table WHERE id = 1;
+ FROM __test_custom_function_table__ WHERE id = 1;
SELECT assert(name = 'sum_fn1', 'Custom function sum_fn1 found in table.')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test adding an existing function name should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
$TRAP$) = 1, 'Should error out for duplicate function name');
/* Test deletion by id where valid table exists */
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'id 2 should exist before deletion!')
- FROM test_custom_function_table WHERE id = 2;
-SELECT delete_custom_function('test_custom_function_table', 2);
+ FROM __test_custom_function_table__ WHERE id = 2;
+SELECT delete_custom_function('__test_custom_function_table__', 2);
SELECT assert(COUNT(id) = 0, 'id 2 should have been deleted!')
- FROM test_custom_function_table WHERE id = 2;
+ FROM __test_custom_function_table__ WHERE id = 2;
/* Test deletion by name where valid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn1');
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn1');
/* Assert id exists before deleting */
SELECT assert(COUNT(id) = 1, 'function name sum_fn1 should exist before deletion!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
-SELECT delete_custom_function('test_custom_function_table', 'sum_fn1');
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
+SELECT delete_custom_function('__test_custom_function_table__', 'sum_fn1');
SELECT assert(COUNT(id) = 0, 'function name sum_fn1 should have been deleted!')
- FROM test_custom_function_table WHERE name = 'sum_fn1';
+ FROM __test_custom_function_table__ WHERE name = 'sum_fn1';
/* Test deleting an already deleted entry should error out */
SELECT assert(MADLIB_SCHEMA.trap_error($TRAP$
- SELECT delete_custom_function('test_custom_function_table', 2);
+ SELECT delete_custom_function('__test_custom_function_table__', 2);
$TRAP$) = 1, 'Should error out for trying to delete an entry that does not exist');
/* Test delete drops the table after deleting last entry*/
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-SELECT delete_custom_function('test_custom_function_table', 1);
-SELECT assert(COUNT(relname) = 0, 'Table test_custom_function_table should have been deleted.')
- FROM pg_class where relname='test_custom_function_table';
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+SELECT delete_custom_function('__test_custom_function_table__', 1);
+SELECT assert(COUNT(relname) = 0, 'Table __test_custom_function_table__ should have been deleted.')
+ FROM pg_class where relname='__test_custom_function_table__';
/* Test deletion where empty table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-DELETE FROM test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+DELETE FROM __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting function in an empty table should generate an exception.');
/* Test deletion where no table exists */
-DROP TABLE IF EXISTS test_custom_function_table;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 1)$$) = 1,
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 1)$$) = 1,
'Deleting a non-existent table should raise exception.');
/* Test where invalid table exists */
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum');
-ALTER TABLE test_custom_function_table DROP COLUMN id;
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('test_custom_function_table', 2)$$) = 1,
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum');
+ALTER TABLE __test_custom_function_table__ DROP COLUMN id;
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT delete_custom_function('__test_custom_function_table__', 2)$$) = 1,
'Deleting an invalid table should generate an exception.');
-SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
+SELECT assert(MADLIB_SCHEMA.trap_error($$SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', 'returns sum')$$) = 1,
'Passing an invalid table to load_custom_function() should raise exception.');
/* Test input validation */
-DROP TABLE IF EXISTS test_custom_function_table;
+DROP TABLE IF EXISTS __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', custom_function_object(), NULL, NULL);
+ SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), NULL, NULL);
$$) = 1, 'Name cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', NULL, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', NULL, 'sum_fn', NULL);
$$) = 1, 'Function object cannot be NULL');
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT load_custom_function('test_custom_function_table', 'invalid_obj'::bytea, 'sum_fn', NULL);
+ SELECT load_custom_function('__test_custom_function_table__', 'invalid_obj'::bytea, 'sum_fn', NULL);
$$) = 1, 'Invalid custom function object');
-SELECT load_custom_function('test_custom_function_table', custom_function_object(), 'sum_fn', NULL);
+SELECT load_custom_function('__test_custom_function_table__', custom_function_object(), 'sum_fn', NULL);
SELECT assert(name IS NOT NULL AND description IS NULL, 'validate name is not NULL.')
- FROM test_custom_function_table;
+ FROM __test_custom_function_table__;
SELECT assert(MADLIB_SCHEMA.trap_error($$
- SELECT delete_custom_function('test_custom_function_table', NULL);
+ SELECT delete_custom_function('__test_custom_function_table__', NULL);
$$) = 1, 'id/name cannot be NULL!');
+
+/* Test top n accuracy */
+
+DROP TABLE IF EXISTS __test_custom_function_table__;
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 3);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7);
+SELECT load_top_n_accuracy_function('__test_custom_function_table__', 7, True);
Review comment:
I see that it prints the following INFO when executing the function :
```
INFO: Keras Custom Function: Added function sparse_top_7_accuracy to __test_custom_function_table__ table
CONTEXT: PL/Python function "load_custom_function"
SQL statement "
SELECT madlib.load_custom_function('__test_custom_function_table__',
__madlib__sparse_top_7_acc_pickled(),
'sparse_top_7_accuracy',
'returns sparse_top_7_accuracy');
```
This comes from the call to `load_custom_function()`. Looking at the above output was just thinking if we should print that out in load_custom_function. Thoughts ?
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_custom_function.py_in
##########
@@ -146,6 +146,43 @@ def update_builtin_metrics(builtin_metrics):
builtin_metrics.append('ce')
return builtin_metrics
+@MinWarning("error")
+def load_top_n_accuracy_function(schema_madlib, object_table, n, is_sparse, **kwargs):
+
+ object_table = quote_ident(object_table)
+ _assert(n > 0,
+ "{0}: For top n accuracy functions n has to be a positive integer.".format(module_name))
+ sparse_prefix = "sparse_" if is_sparse else ""
+
+ sql = """
+ CREATE OR REPLACE FUNCTION __madlib__{sparse_prefix}top_{n}_acc_pickled()
Review comment:
Will the user be constraint from using this utility based on their privileges ? Since we are trying to create a function inside the utility, if a user doesn't have privileges to create UDF's, I suppose this will error out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org