You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by GitBox <gi...@apache.org> on 2019/10/29 16:59:32 UTC
[GitHub] [madlib] fmcquillan99 commented on issue #451: DL: Update evaluate
and predict for multi model outputs
fmcquillan99 commented on issue #451: DL: Update evaluate and predict for multi model outputs
URL: https://github.com/apache/madlib/pull/451#issuecomment-547525859
(1)
- iris data
- 2 MLP models (1 and 2 hidden layers)
```
SELECT * FROM mst_table ORDER BY mst_key;
mst_key | model_id | compile_params | fit_params
---------+----------+---------------------------------------------------------------------------------+-----------------------
1 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=4,epochs=1
2 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=8,epochs=1
3 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=4,epochs=1
4 | 1 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=8,epochs=1
5 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=4,epochs=1
6 | 1 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=8,epochs=1
7 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=4,epochs=1
8 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy'] | batch_size=8,epochs=1
9 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=4,epochs=1
10 | 2 | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy'] | batch_size=8,epochs=1
11 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=4,epochs=1
12 | 2 | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy'] | batch_size=8,epochs=1
(12 rows)
```
Train
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
10, -- num_iterations
0 -- use gpus
);
SELECT * FROM iris_multi_model_info ORDER BY training_metrics_final DESC;
-[ RECORD 1 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 3
model_id | 1
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.15962290763855,0.171411037445068,0.154904842376709,0.151599884033203,0.160181999206543,0.152440071105957,0.223697900772095,0.156054973602295
,0.202955007553101,0.152594089508057}
metrics_type | {accuracy}
training_metrics_final | 0.975000023842
training_loss_final | 0.0928164944053
training_metrics | {0.675000011920929,0.875,0.725000023841858,0.975000023841858,0.966666638851166,0.983333349227905,0.975000023841858,0.975000023841858,0.88333332
5386047,0.975000023841858}
training_loss | {0.520763278007507,0.379715353250504,0.406634271144867,0.182054653763771,0.14105860888958,0.108805403113365,0.109423443675041,0.101339735090733
,0.215542033314705,0.0928164944052696}
validation_metrics_final | 0.933333337307
validation_loss_final | 0.196297511458
validation_metrics | {0.733333349227905,0.899999976158142,0.766666650772095,0.933333337306976,0.899999976158142,0.966666638851166,0.933333337306976,0.93333333730697
6,0.800000011920929,0.933333337306976}
validation_loss | {0.487521052360535,0.348948538303375,0.474558025598526,0.232045486569405,0.22162064909935,0.155132472515106,0.202082633972168,0.196883022785187
,0.382103145122528,0.196297511458397}
-[ RECORD 2 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.183917045593262,0.373061180114746,0.182140111923218,0.177560091018677,0.192739009857178,0.177842140197754,0.201126098632812,0.17956304550170
9,0.437918901443481,0.184372901916504}
metrics_type | {accuracy}
training_metrics_final | 0.966666638851
training_loss_final | 0.182971566916
training_metrics | {0.658333361148834,0.658333361148834,0.858333349227905,0.691666662693024,0.983333349227905,0.958333313465118,0.800000011920929,0.91666668653488
2,0.958333313465118,0.966666638851166}
training_loss | {0.839326024055481,0.552537620067596,0.427368670701981,0.395185142755508,0.339115768671036,0.288732022047043,0.310250341892242,0.31521281599998
5,0.203457966446877,0.182971566915512}
validation_metrics_final | 0.933333337307
validation_loss_final | 0.21536397934
validation_metrics | {0.699999988079071,0.699999988079071,0.766666650772095,0.733333349227905,0.899999976158142,0.866666674613953,0.766666650772095,0.89999997615814
2,0.833333313465118,0.933333337306976}
validation_loss | {0.841938138008118,0.521961748600006,0.390463948249817,0.383590310811996,0.336574584245682,0.328944146633148,0.382724314928055,0.32695025205612
2,0.299405694007874,0.2153639793396}
-[ RECORD 3 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 4
model_id | 1
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.158810138702393,0.171705007553101,0.157279968261719,0.149960994720459,0.161519050598145,0.151033163070679,0.167169094085693,0.15194702148437
5,0.181770086288452,0.160482168197632}
metrics_type | {accuracy}
training_metrics_final | 0.933333337307
training_loss_final | 0.109170868993
training_metrics | {0.658333361148834,0.933333337306976,0.983333349227905,0.933333337306976,0.933333337306976,0.983333349227905,0.983333349227905,0.98333334922790
5,0.991666674613953,0.933333337306976}
training_loss | {0.703937888145447,0.404870539903641,0.223930105566978,0.183146089315414,0.137469902634621,0.0971243679523468,0.0983915776014328,0.095256045460
701,0.0844478830695152,0.109170868992805}
validation_metrics_final | 0.933333337307
validation_loss_final | 0.140566334128
validation_metrics | {0.699999988079071,0.899999976158142,0.899999976158142,0.899999976158142,0.933333337306976,0.899999976158142,0.899999976158142,0.89999997615814
2,0.966666638851166,0.933333337306976}
validation_loss | {0.648390173912048,0.386223673820496,0.268760353326797,0.211181059479713,0.171986550092697,0.192580834031105,0.217913091182709,0.21975873410701
8,0.143908932805061,0.14056633412838}
-[ RECORD 4 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 9
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.184342861175537,0.186679124832153,0.178445100784302,0.198610067367554,0.179869174957275,0.176558017730713,0.191009998321533,0.18026208877563
5,0.191850900650024,0.180068969726562}
metrics_type | {accuracy}
training_metrics_final | 0.933333337307
training_loss_final | 0.132100492716
training_metrics | {0.658333361148834,0.966666638851166,0.916666686534882,0.899999976158142,0.708333313465118,0.966666638851166,0.975000023841858,0.875,0.97500002
3841858,0.933333337306976}
training_loss | {0.645366847515106,0.232219696044922,0.187613621354103,0.240386798977852,0.561961591243744,0.117926396429539,0.0994494929909706,0.2343605309724
81,0.0802617892622948,0.132100492715836}
validation_metrics_final | 0.899999976158
validation_loss_final | 0.159418046474
validation_metrics | {0.699999988079071,0.866666674613953,0.833333313465118,0.800000011920929,0.733333349227905,0.933333337306976,0.933333337306976,0.86666667461395
3,0.966666638851166,0.899999976158142}
validation_loss | {0.614455163478851,0.268589198589325,0.27994567155838,0.367912858724594,0.662781298160553,0.164628267288208,0.192523822188377,0.292809307575226
,0.132961437106133,0.159418046474457}
-[ RECORD 5 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 5
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.177564144134521,0.156223058700562,0.154070854187012,0.161600112915039,0.150278091430664,0.164193868637085,0.155445098876953,0.17075300216674
8,0.155435085296631,0.153830051422119}
metrics_type | {accuracy}
training_metrics_final | 0.899999976158
training_loss_final | 0.44444206357
training_metrics | {0.708333313465118,0.658333361148834,0.658333361148834,0.658333361148834,0.675000011920929,0.708333313465118,0.708333313465118,0.75833332538604
7,0.766666650772095,0.899999976158142}
training_loss | {0.771447777748108,0.682944893836975,0.630482733249664,0.594800889492035,0.564319252967834,0.533434450626373,0.50351870059967,0.481856644153595
,0.46294954419136,0.444442063570023}
validation_metrics_final | 0.833333313465
validation_loss_final | 0.424488693476
validation_metrics | {0.766666650772095,0.699999988079071,0.699999988079071,0.699999988079071,0.733333349227905,0.733333349227905,0.733333349227905,0.73333334922790
5,0.733333349227905,0.833333313465118}
validation_loss | {0.759826004505157,0.662849307060242,0.604207634925842,0.569091618061066,0.536879122257233,0.504582524299622,0.477831065654755,0.45702105760574
3,0.440479785203934,0.424488693475723}
-[ RECORD 6 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.16278600692749,0.154292821884155,0.161294937133789,0.153645992279053,0.164122104644775,0.152410984039307,0.396697998046875,0.154489994049072
,0.15544605255127,0.164608001708984}
metrics_type | {accuracy}
training_metrics_final | 0.783333361149
training_loss_final | 0.442915290594
training_metrics | {0.349999994039536,0.516666650772095,0.975000023841858,0.791666686534882,0.883333325386047,0.949999988079071,0.966666638851166,0.91666668653488
2,0.966666638851166,0.783333361148834}
training_loss | {1.17399632930756,0.733614087104797,0.312711358070374,0.494298249483109,0.256883263587952,0.230874612927437,0.164221689105034,0.264884918928146
,0.129683092236519,0.442915290594101}
validation_metrics_final | 0.766666650772
validation_loss_final | 0.658641517162
validation_metrics | {0.300000011920929,0.433333337306976,0.933333337306976,0.866666674613953,0.899999976158142,0.966666638851166,1,0.866666674613953,0.899999976158
142,0.766666650772095}
validation_loss | {1.40960335731506,0.879667580127716,0.289381295442581,0.503752708435059,0.241700142621994,0.224781185388565,0.176762118935585,0.270557969808578
,0.220981314778328,0.658641517162323}
-[ RECORD 7 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 7
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.375895977020264,0.179503917694092,0.181375026702881,0.18864917755127,0.181995868682861,0.197070121765137,0.178755044937134,0.419157028198242
,0.185692071914673,0.178522109985352}
metrics_type | {accuracy}
training_metrics_final | 0.75
training_loss_final | 0.37944367528
training_metrics | {0.658333361148834,0.658333361148834,0.958333313465118,0.658333361148834,0.616666674613953,0.416666656732559,0.875,0.966666638851166,0.98333334
9227905,0.75}
training_loss | {0.564021706581116,0.656923830509186,0.410611629486084,0.657700955867767,0.783278226852417,0.938268542289734,0.240705713629723,0.19149164855480
2,0.188495144248009,0.379443675279617}
validation_metrics_final | 0.733333349228
validation_loss_final | 0.440304368734
validation_metrics | {0.699999988079071,0.699999988079071,0.933333337306976,0.699999988079071,0.633333325386047,0.366666674613953,0.766666650772095,0.96666663885116
6,0.899999976158142,0.733333349227905}
validation_loss | {0.518877744674683,0.581225216388702,0.369613200426102,0.575586378574371,0.760259926319122,1.11823546886444,0.337272256612778,0.201485335826874
,0.231084078550339,0.44030436873436}
-[ RECORD 8 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 11
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.185693979263306,0.176743030548096,0.208944082260132,0.177680969238281,0.188605070114136,0.179240942001343,0.469206094741821,0.18075394630432
1,0.182642936706543,0.18809700012207}
metrics_type | {accuracy}
training_metrics_final | 0.716666638851
training_loss_final | 0.49047088623
training_metrics | {0.341666668653488,0.341666668653488,0.366666674613953,0.883333325386047,0.783333361148834,0.658333361148834,0.658333361148834,0.66666668653488
2,0.866666674613953,0.716666638851166}
training_loss | {1.10631430149078,1.00225436687469,0.952353298664093,0.879558265209198,0.816462278366089,0.759344756603241,0.688487946987152,0.61039125919342,0
.543096899986267,0.490470886230469}
validation_metrics_final | 0.766666650772
validation_loss_final | 0.471391111612
validation_metrics | {0.300000011920929,0.300000011920929,0.333333343267441,0.766666650772095,0.766666650772095,0.699999988079071,0.733333349227905,0.73333334922790
5,0.766666650772095,0.766666650772095}
validation_loss | {1.18413507938385,1.03528475761414,0.984834551811218,0.890941798686981,0.828823924064636,0.763210475444794,0.687682032585144,0.596106350421906,
0.527397036552429,0.47139111161232}
-[ RECORD 9 ]------------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 1
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.155688047409058,0.162626981735229,0.157097101211548,0.175328969955444,0.151535987854004,0.382455110549927,0.155317068099976,0.15313291549682
6,0.161896944046021,0.154218196868896}
metrics_type | {accuracy}
training_metrics_final | 0.658333361149
training_loss_final | 0.506188511848
training_metrics | {0.658333361148834,0.583333313465118,0.658333361148834,0.808333337306976,0.583333313465118,0.983333349227905,0.975000023841858,0.85833334922790
5,0.666666686534882,0.658333361148834}
training_loss | {0.634293258190155,0.62472277879715,0.491173326969147,0.325925976037979,0.834470629692078,0.270386427640915,0.137209206819534,0.237895086407661
,0.488089859485626,0.50618851184845}
validation_metrics_final | 0.699999988079
validation_loss_final | 0.437518566847
validation_metrics | {0.699999988079071,0.633333325386047,0.699999988079071,0.766666650772095,0.633333325386047,0.933333337306976,0.933333337306976,0.89999997615814
2,0.699999988079071,0.699999988079071}
validation_loss | {0.570496320724487,0.620569884777069,0.43601530790329,0.389920294284821,1.09760916233063,0.271371573209763,0.166262924671173,0.227906361222267,
0.429942190647125,0.437518566846848}
-[ RECORD 10 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 6
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.175873041152954,0.155245065689087,0.154464960098267,0.171594858169556,0.157553195953369,0.162687063217163,0.155516862869263,0.16730904579162
6,0.156662940979004,0.396471977233887}
metrics_type | {accuracy}
training_metrics_final | 0.508333325386
training_loss_final | 0.915417551994
training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.308333337306976,0.324999988079071,0.341666668653488,0.34166666865348
8,0.358333319425583,0.508333325386047}
training_loss | {2.13231420516968,1.79814553260803,1.51453709602356,1.26933741569519,1.15679371356964,1.07199251651764,1.01928377151489,0.978273630142212,0.942
150175571442,0.915417551994324}
validation_metrics_final | 0.600000023842
validation_loss_final | 0.93969720602
validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.333333343267441,0.333333343267441,0.300000011920929,0.30000001192092
9,0.333333343267441,0.600000023841858}
validation_loss | {2.51236271858215,2.09125304222107,1.73099553585052,1.41019690036774,1.26361191272736,1.14860534667969,1.07588887214661,1.01970684528351,0.9719
77055072784,0.939697206020355}
-[ RECORD 11 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 12
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.184770822525024,0.191632032394409,0.178493022918701,0.391031980514526,0.181709051132202,0.17969799041748,0.191845893859863,0.179744958877563
,0.199568033218384,0.178317070007324}
metrics_type | {accuracy}
training_metrics_final | 0.341666668653
training_loss_final | 1.09104967117
training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.34166666865348
8,0.341666668653488,0.341666668653488}
training_loss | {3.08191418647766,2.32425045967102,1.79031324386597,1.4521062374115,1.26406502723694,1.19002187252045,1.14052081108093,1.11407339572906,1.10518
69392395,1.0910496711731}
validation_metrics_final | 0.300000011921
validation_loss_final | 1.09330034256
validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.30000001192092
9,0.300000011920929,0.300000011920929}
validation_loss | {3.12394428253174,2.34826755523682,1.7957478761673,1.44716620445251,1.25600838661194,1.18111801147461,1.13339793682098,1.11054337024689,1.10561
001300812,1.09330034255981}
-[ RECORD 12 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
mst_key | 8
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.270622968673706,0.18524694442749,0.178076028823853,0.200053930282593,0.178044080734253,0.198669195175171,0.180491924285889,0.182524919509888
,0.186138868331909,0.175627946853638}
metrics_type | {accuracy}
training_metrics_final | 0.341666668653
training_loss_final | 1.09889221191
training_metrics | {0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.341666668653488,0.34166666865348
8,0.316666662693024,0.341666668653488}
training_loss | {1.09904658794403,1.10012602806091,1.10846817493439,1.09847903251648,1.10446727275848,1.10296499729156,1.097984790802,1.10039520263672,1.103934
88407135,1.09889221191406}
validation_metrics_final | 0.300000011921
validation_loss_final | 1.10708355904
validation_metrics | {0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.300000011920929,0.30000001192092
9,0.400000005960464,0.300000011920929}
validation_loss | {1.10612213611603,1.11198484897614,1.12145566940308,1.10108768939972,1.09653890132904,1.12743330001831,1.10469746589661,1.11833906173706,1.0917
9925918579,1.10708355903625}
```
seems OK
(2)
evaluate on test data
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model
'iris_test_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
9 -- mst_key to use
);
SELECT * FROM iris_validate;
loss | metric | metrics_type
-------------------+-------------------+--------------
0.159418046474457 | 0.899999976158142 | {accuracy}
(1 row)
```
seems OK, matches final acc and loss for model 9 on validation data during training
(3)
evaluate on training data
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model
'iris_train_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
9 -- mst_key to use
);
SELECT * FROM iris_validate;
loss | metric | metrics_type
-------------------+-------------------+--------------
0.132100492715836 | 0.933333337306976 | {accuracy}
(1 row)
```
seems OK, matches final acc and loss for model 9 on training data during training
(4)
predict on training data
```
DROP TABLE IF EXISTS iris_predict;
SELECT madlib.madlib_keras_predict('iris_multi_model', -- model
'iris_train', -- train_table
'id', -- id column
'attributes', -- independent var
'iris_predict', -- output table
'response', -- prediction type
NULL, -- use gpus
9 -- mst_key to use
);
SELECT COUNT(*) FROM iris_predict JOIN iris_train USING (id)
WHERE iris_predict.estimated_class_text != iris_train.class_text;
count
-------
34
```
error %
```
SELECT round(count(*)*100/(150*0.8),2) as train_accuracy_percent from
(select iris_train.class_text as actual, iris_predict.estimated_class_text as estimated
from iris_predict inner join iris_train
on iris_train.id=iris_predict.id) q
WHERE q.actual=q.estimated;
train_accuracy_percent
------------------------
71.67
```
Why is this not 0.933333337306976 ???
(5)
predict on test data
```
DROP TABLE IF EXISTS iris_predict;
SELECT madlib.madlib_keras_predict('iris_multi_model', -- model
'iris_test', -- test_table
'id', -- id column
'attributes', -- independent var
'iris_predict', -- output table
'response', -- prediction type
NULL, -- use gpus
9 -- mst_key to use
);
SELECT COUNT(*) FROM iris_predict JOIN iris_test USING (id)
WHERE iris_predict.estimated_class_text != iris_test.class_text;
count
-------
7
(1 row)
```
as a %
```
SELECT round(count(*)*100/(150*0.2),2) as test_accuracy_percent from
(select iris_test.class_text as actual, iris_predict.estimated_class_text as estimated
from iris_predict inner join iris_test
on iris_test.id=iris_predict.id) q
WHERE q.actual=q.estimated;
test_accuracy_percent
-----------------------
76.67
(1 row)
```
Why is this not 0.899999976158142 ???
(6)
predict is giving warning errors which should be suppressed
```
WARNING: column "attributes_shape" does not exist
CONTEXT: PL/Python function "madlib_keras_predict"
madlib_keras_predict
----------------------
```
Please check other functions in this module to ensure the right level of console reporting is set.
(7)
pick wrong model
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model
'iris_test_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
999 -- mst_key to use
);
ERROR: plpy.Error: madlib_keras_evaluate error: Input table '__madlib_temp_tmp_view_summary39503791_1572367130_16846764__' is empty! (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_evaluate", line 21, in <module>
return madlib_keras.evaluate(**globals())
PL/Python function "madlib_keras_evaluate", line 577, in evaluate
PL/Python function "madlib_keras_evaluate", line 628, in validate_evaluate
PL/Python function "madlib_keras_evaluate", line 59, in validate_predict_evaluate_tables
PL/Python function "madlib_keras_evaluate", line 218, in _validate_model_summary_tbl
PL/Python function "madlib_keras_evaluate", line 678, in input_tbl_valid
PL/Python function "madlib_keras_evaluate"
```
This error is misleading - should say that 999 is not a valid mst_key or model or whatever.
(8)
input table does not exist
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model999', -- model
'iris_test_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
9 -- mst_key to use
);
ERROR: NameError: global name 'module_name' is not defined (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_evaluate", line 21, in <module>
return madlib_keras.evaluate(**globals())
PL/Python function "madlib_keras_evaluate", line 575, in evaluate
PL/Python function "madlib_keras_evaluate", line 237, in create_summary_view
PL/Python function "madlib_keras_evaluate"
```
This error is misleading - should say source table `iris_multi_model999` does not exist
(9)
please do a general check that we are trapping input errors in the usual way for fit, predict and evaluate. I did not test them all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services