You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by GitBox <gi...@apache.org> on 2020/09/10 23:04:17 UTC
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #513: DL: [AutoML] Add support for 'diagonal' Hyperband optimized for MPP
fmcquillan99 edited a comment on pull request #513:
URL: https://github.com/apache/madlib/pull/513#issuecomment-690776040
iris dataset
(1)
schedule:
```
s | i | n_i | r_i
---+---+-----+-----
2 | 0 | 9 | 1
2 | 1 | 3 | 3
2 | 2 | 1 | 9
1 | 0 | 3 | 3
1 | 1 | 1 | 9
0 | 0 | 3 | 9
(6 rows)
```
train:
```
DROP TABLE IF EXISTS automl_output, automl_output_info, automl_output_summary, automl_mst_table, automl_mst_table_summary;
SELECT madlib.madlib_keras_automl('iris_train_packed',
'automl_output',
'model_arch_library_iris',
'automl_mst_table',
ARRAY[1],
$${'loss': ['categorical_crossentropy'], 'optimizer_params_list': [ {'optimizer': ['Adam'],'lr': [0.00999, 0.01001, 'log']}], 'metrics':['accuracy']}$$,
$${'batch_size': [8], 'epochs': [1]}$$,
'hyperband',
'R=9, eta=3, skip_last=0',
42, -- random state
NULL, -- object table
FALSE, -- use GPUs
'iris_test_packed', -- validation table
1, -- metrics compute freq
'a name is a name', -- name
'a descr is a descr'); -- descr
```
produces:
```
INFO: *** Diagonally evaluating 9 configs under bracket=2 & round=0 with 1 iterations ***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 8.45693087578 sec
DETAIL:
Training set after iteration 1:
mst_key=2: metric=0.933333337307, loss=0.538550555706
mst_key=8: metric=0.316666662693, loss=1.03528738022
mst_key=3: metric=0.675000011921, loss=0.833593308926
mst_key=6: metric=0.658333361149, loss=0.825735092163
mst_key=1: metric=0.808333337307, loss=0.872115910053
mst_key=7: metric=0.625, loss=0.974291205406
mst_key=9: metric=0.658333361149, loss=0.830321311951
mst_key=4: metric=0.658333361149, loss=0.885693848133
mst_key=5: metric=0.341666668653, loss=1.09756851196
Validation set after iteration 1:
mst_key=2: metric=0.966666638851, loss=0.534965932369
mst_key=8: metric=0.366666674614, loss=1.03363573551
mst_key=3: metric=0.633333325386, loss=0.872259676456
mst_key=6: metric=0.699999988079, loss=0.799027740955
mst_key=1: metric=0.833333313465, loss=0.866640985012
mst_key=7: metric=0.5, loss=0.974297106266
mst_key=9: metric=0.699999988079, loss=0.829889714718
mst_key=4: metric=0.699999988079, loss=0.885949194431
mst_key=5: metric=0.300000011921, loss=1.10424804688
CONTEXT: PL/Python function "madlib_keras_automl"
INFO: *** Diagonally evaluating 3 configs under bracket=2 & round=1, 3 configs under bracket=1 & round=0 with 3 iterations ***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 4.89221119881 sec
DETAIL:
Training set after iteration 1:
mst_key=9: metric=0.658333361149, loss=0.530654907227
mst_key=6: metric=0.774999976158, loss=0.531277298927
mst_key=11: metric=0.691666662693, loss=0.861804306507
mst_key=2: metric=0.716666638851, loss=0.426045030355
mst_key=12: metric=0.608333349228, loss=0.984588444233
mst_key=10: metric=0.324999988079, loss=1.17993462086
Validation set after iteration 1:
mst_key=9: metric=0.699999988079, loss=0.514933288097
mst_key=6: metric=0.800000011921, loss=0.519642591476
mst_key=11: metric=0.666666686535, loss=0.861222624779
mst_key=2: metric=0.733333349228, loss=0.385236173868
mst_key=12: metric=0.699999988079, loss=0.961848556995
mst_key=10: metric=0.366666674614, loss=1.16651546955
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 5.0256319046 sec
DETAIL:
Training set after iteration 2:
mst_key=9: metric=0.966666638851, loss=0.432032138109
mst_key=6: metric=0.908333361149, loss=0.361528068781
mst_key=11: metric=0.858333349228, loss=0.686981141567
mst_key=2: metric=0.758333325386, loss=0.373835682869
mst_key=12: metric=0.600000023842, loss=0.89406478405
mst_key=10: metric=0.341666668653, loss=1.13211238384
Validation set after iteration 2:
mst_key=9: metric=0.933333337307, loss=0.442799389362
mst_key=6: metric=0.933333337307, loss=0.350269824266
mst_key=11: metric=0.833333313465, loss=0.694237530231
mst_key=2: metric=0.899999976158, loss=0.325422286987
mst_key=12: metric=0.699999988079, loss=0.868638038635
mst_key=10: metric=0.300000011921, loss=1.14590144157
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 3: 4.99592685699 sec
DETAIL:
Training set after iteration 3:
mst_key=9: metric=0.699999988079, loss=0.403143405914
mst_key=6: metric=0.949999988079, loss=0.292406588793
mst_key=11: metric=0.949999988079, loss=0.614805757999
mst_key=2: metric=0.658333361149, loss=0.662374198437
mst_key=12: metric=0.47499999404, loss=0.713567197323
mst_key=10: metric=0.40000000596, loss=1.08757543564
Validation set after iteration 3:
mst_key=9: metric=0.733333349228, loss=0.365277469158
mst_key=6: metric=0.899999976158, loss=0.303717941046
mst_key=11: metric=0.966666638851, loss=0.615865588188
mst_key=2: metric=0.633333325386, loss=0.758127868176
mst_key=12: metric=0.566666662693, loss=0.689917981625
mst_key=10: metric=0.300000011921, loss=1.09673452377
CONTEXT: PL/Python function "madlib_keras_automl"
INFO: *** Diagonally evaluating 1 configs under bracket=2 & round=2, 1 configs under bracket=1 & round=1, 3 configs under bracket=0 & round=0 with 9 iterations ***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 4.01545906067 sec
DETAIL:
Training set after iteration 1:
mst_key=11: metric=0.949999988079, loss=0.535947144032
mst_key=13: metric=0.658333361149, loss=0.576903045177
mst_key=6: metric=0.725000023842, loss=0.413865655661
mst_key=15: metric=0.566666662693, loss=0.821544110775
mst_key=14: metric=0.608333349228, loss=1.00213348866
Validation set after iteration 1:
mst_key=11: metric=0.866666674614, loss=0.556911349297
mst_key=13: metric=0.699999988079, loss=0.577660501003
mst_key=6: metric=0.666666686535, loss=0.506598353386
mst_key=15: metric=0.600000023842, loss=0.782476782799
mst_key=14: metric=0.566666662693, loss=1.00861167908
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 4.0810611248 sec
DETAIL:
Training set after iteration 2:
mst_key=11: metric=0.958333313465, loss=0.461322844028
mst_key=13: metric=0.658333361149, loss=0.488560408354
mst_key=6: metric=0.824999988079, loss=0.323733448982
mst_key=15: metric=0.875, loss=0.656310975552
mst_key=14: metric=0.658333361149, loss=0.641188919544
Validation set after iteration 2:
mst_key=11: metric=0.866666674614, loss=0.486970752478
mst_key=13: metric=0.699999988079, loss=0.468940109015
mst_key=6: metric=0.733333349228, loss=0.393083393574
mst_key=15: metric=0.800000011921, loss=0.679034292698
mst_key=14: metric=0.699999988079, loss=0.636842608452
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 3: 3.9402680397 sec
DETAIL:
Training set after iteration 3:
mst_key=11: metric=0.841666638851, loss=0.428030073643
mst_key=13: metric=0.933333337307, loss=0.341508597136
mst_key=6: metric=0.933333337307, loss=0.217096969485
mst_key=15: metric=0.941666662693, loss=0.388799399137
mst_key=14: metric=0.916666686535, loss=0.44380658865
Validation set after iteration 3:
mst_key=11: metric=0.733333349228, loss=0.495877951384
mst_key=13: metric=0.966666638851, loss=0.35888004303
mst_key=6: metric=0.933333337307, loss=0.196621760726
mst_key=15: metric=0.866666674614, loss=0.401055067778
mst_key=14: metric=0.933333337307, loss=0.44562998414
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 4: 4.27783608437 sec
DETAIL:
Training set after iteration 4:
mst_key=11: metric=0.899999976158, loss=0.348980903625
mst_key=13: metric=0.949999988079, loss=0.273842066526
mst_key=6: metric=0.824999988079, loss=0.307050615549
mst_key=15: metric=0.941666662693, loss=0.25471162796
mst_key=14: metric=0.966666638851, loss=0.266296118498
Validation set after iteration 4:
mst_key=11: metric=0.833333313465, loss=0.420216292143
mst_key=13: metric=0.866666674614, loss=0.317742973566
mst_key=6: metric=0.899999976158, loss=0.230505168438
mst_key=15: metric=0.899999976158, loss=0.281882911921
mst_key=14: metric=0.966666638851, loss=0.275415152311
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 5: 4.06583809853 sec
DETAIL:
Training set after iteration 5:
mst_key=11: metric=0.958333313465, loss=0.315752029419
mst_key=13: metric=0.708333313465, loss=0.4172565341
mst_key=6: metric=0.958333313465, loss=0.151342079043
mst_key=15: metric=0.975000023842, loss=0.179173886776
mst_key=14: metric=0.983333349228, loss=0.177039191127
Validation set after iteration 5:
mst_key=11: metric=0.866666674614, loss=0.342488378286
mst_key=13: metric=0.633333325386, loss=0.558800399303
mst_key=6: metric=0.966666638851, loss=0.153643578291
mst_key=15: metric=1.0, loss=0.174838155508
mst_key=14: metric=0.966666638851, loss=0.192256584764
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 6: 4.02019715309 sec
DETAIL:
Training set after iteration 6:
mst_key=11: metric=0.883333325386, loss=0.296552926302
mst_key=13: metric=0.899999976158, loss=0.248627394438
mst_key=6: metric=0.858333349228, loss=0.285588264465
mst_key=15: metric=0.949999988079, loss=0.167279005051
mst_key=14: metric=0.899999976158, loss=0.227244228125
Validation set after iteration 6:
mst_key=11: metric=0.833333313465, loss=0.401087760925
mst_key=13: metric=0.833333313465, loss=0.321891099215
mst_key=6: metric=0.733333349228, loss=0.421591818333
mst_key=15: metric=0.966666638851, loss=0.138948410749
mst_key=14: metric=0.800000011921, loss=0.34265729785
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 7: 4.27092909813 sec
DETAIL:
Training set after iteration 7:
mst_key=11: metric=0.891666650772, loss=0.248297154903
mst_key=13: metric=0.891666650772, loss=0.24527259171
mst_key=6: metric=0.949999988079, loss=0.12675113976
mst_key=15: metric=0.916666686535, loss=0.215490117669
mst_key=14: metric=0.958333313465, loss=0.127076357603
Validation set after iteration 7:
mst_key=11: metric=0.866666674614, loss=0.335224747658
mst_key=13: metric=0.766666650772, loss=0.349867284298
mst_key=6: metric=1.0, loss=0.120815463364
mst_key=15: metric=0.933333337307, loss=0.161028474569
mst_key=14: metric=0.899999976158, loss=0.196913182735
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 8: 4.16106319427 sec
DETAIL:
Training set after iteration 8:
mst_key=11: metric=0.958333313465, loss=0.21432980895
mst_key=13: metric=0.75, loss=0.440660595894
mst_key=6: metric=0.791666686535, loss=0.416923582554
mst_key=15: metric=0.933333337307, loss=0.161955311894
mst_key=14: metric=0.908333361149, loss=0.182602509856
Validation set after iteration 8:
mst_key=11: metric=0.866666674614, loss=0.264829039574
mst_key=13: metric=0.699999988079, loss=0.644149899483
mst_key=6: metric=0.899999976158, loss=0.281583458185
mst_key=15: metric=0.866666674614, loss=0.226034641266
mst_key=14: metric=0.866666674614, loss=0.33136048913
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 9: 4.84911680222 sec
DETAIL:
Training set after iteration 9:
mst_key=11: metric=0.975000023842, loss=0.221068292856
mst_key=13: metric=0.983333349228, loss=0.138952031732
mst_key=6: metric=0.816666662693, loss=0.403530687094
mst_key=15: metric=0.941666662693, loss=0.168528050184
mst_key=14: metric=0.958333313465, loss=0.14608065784
Validation set after iteration 9:
mst_key=11: metric=0.966666638851, loss=0.236671224236
mst_key=13: metric=0.966666638851, loss=0.161113128066
mst_key=6: metric=0.899999976158, loss=0.267582565546
mst_key=15: metric=0.966666638851, loss=0.120766565204
mst_key=14: metric=0.966666638851, loss=0.127617403865
CONTEXT: PL/Python function "madlib_keras_automl"
madlib_keras_automl
---------------------
(1 row)
Time: 93722.857 ms
```
e.g., for bracket 2
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org