You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by GitBox <gi...@apache.org> on 2020/09/22 16:45:00 UTC
[GitHub] [madlib] Advitya17 opened a new pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Advitya17 opened a new pull request #518:
URL: https://github.com/apache/madlib/pull/518
JIRA: MADLIB-1453
We expand AutoML capabilities in MADlib by adding support for Hyperopt, another hyperparameter optimization method to minimize losses with awkward search spaces. The user can declaratively specify the automl method name with the related automl params (with all other argument declarations staying the same across different AutoML methods) and our API handles the exploration and execution components with the algorithm workload info displayed to the user.
We implement Hyperopt for Massively Parallel Processing (MPP) to perform model selection on top of our existing Model hOpper (MOP) infrastructure for model training/evaluation. Our API currently offers declarative support for Random Search and Tree-structured Parzen Estimator (TPE), a Bayesian Optimization like approach.
This PR also includes:
1. Code re-factor to add additional classes (inheriting from the AutoML super class) for any future AutoML methods.
2. 'best-so-far' console info for each AutoML method currently supported (i.e. Hyperband and Hyperopt)
3. Disabling automl method argument prefixing (while still retaining case insensitivity).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-715643042
(100)
error msg
```
madlib=# SELECT madlib.madlib_keras_automl('iris_train_packed', -- source table
madlib(# 'automl_output', -- model output table
madlib(# 'model_arch_library', -- model architecture table
madlib(# 'automl_mst_table', -- model selection output table
madlib(# ARRAY[1,2], -- model IDs
madlib(# $${
madlib$# 'loss': ['categorical_crossentropy'],
madlib$# 'optimizer_params_list': [
madlib$# {'optimizer': ['Adam'],'lr': [0.001, 0.1, 'log']},
madlib$# {'optimizer': ['RMSprop'],'lr': [0.001, 0.1, 'log']}
madlib$# ],
madlib$# 'metrics': ['accuracy']
madlib$# } $$, -- compile param grid
madlib(# $${'batch_size': [4, 8], 'epochs': [1]}$$, -- fit params grid
madlib(# 'hyperband', -- autoML method
madlib(# 'num_configs=6, num_iterations=3, algorithm=rand', -- autoML params
madlib(# NULL, -- random state
madlib(# NULL, -- object table
madlib(# FALSE, -- use GPUs
madlib(# 'iris_test_packed', -- validation table
madlib(# 1, -- metrics compute freq
madlib(# NULL, -- name
madlib(# NULL); -- descr
ERROR: ValueError: invalid literal for int() with base 10: 'rand' (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_automl", line 23, in <module>
schedule_loader = madlib_keras_automl_hyperband.AutoMLHyperband(**globals())
PL/Python function "madlib_keras_automl", line 164, in __init__
PL/Python function "madlib_keras_automl", line 174, in validate_and_define_inputs
PL/Python function "madlib_keras_automl"
```
this message confused me for awhile till I realized I asked for hyperband but gave hyperopt params accidentally
(101)
error msg
```
madlib=# SELECT madlib.madlib_keras_automl('iris_train_packed', -- source table
madlib(# 'automl_output', -- model output table
madlib(# 'model_arch_library', -- model architecture table
madlib(# 'automl_mst_table', -- model selection output table
madlib(# ARRAY[1,2], -- model IDs
madlib(# $${
madlib$# 'loss': ['categorical_crossentropy'],
madlib$# 'optimizer_params_list': [
madlib$# {'optimizer': ['Adam'],'lr': [0.001, 0.1, 'log']},
madlib$# {'optimizer': ['RMSprop'],'lr': [0.001, 0.1, 'log']}
madlib$# ],
madlib$# 'metrics': ['accuracy']
madlib$# } $$, -- compile param grid
madlib(# $${'batch_size': [4, 8], 'epochs': [1]}$$, -- fit params grid
madlib(# 'hyperopt', -- autoML method
madlib(# 'R=9, eta=3, skip_last=0', -- autoML params
madlib(# NULL, -- random state
madlib(# NULL, -- object table
madlib(# FALSE, -- use GPUs
madlib(# 'iris_test_packed', -- validation table
madlib(# 1, -- metrics compute freq
madlib(# NULL, -- name
madlib(# NULL); -- descr
ERROR: plpy.Error: madlib_keras_automl: skip_last is an invalid automl param (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_automl", line 45, in <module>
schedule_loader = madlib_keras_automl_hyperopt.AutoMLHyperopt(**globals())
PL/Python function "madlib_keras_automl", line 69, in __init__
PL/Python function "madlib_keras_automl", line 115, in validate_and_define_inputs
PL/Python function "madlib_keras_automl"
```
likewise this could be improved if I asked for hyperopt but gave hyperband params accidentally. Because skip_last is valid but only for hyberband.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493744195
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
Review comment:
I don't think we should calculate 2**31 every time we run this. Setting an INT_MAX constant would be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707399640
> mst_key | metrics_elapsed_time
> ---------+------------------------------------------------------
> 1 | {627.882262945175,1264.80915999413,1896.74943590164}
> 2 | {614.573877811432,1251.1022670269,1883.14707994461}
> 3 | {638.407788038254,1286.73397421837,1939.58357810974}
> 4 | {624.925609111786,1273.55782914162,1925.82674407959}
> 5 | {650.984225034714,1307.78145503998,1962.8148560524}
> 6 | {637.914520978928,1294.23532605171,1949.12843108177}
> 7 | {651.188719987869,1301.32990193367,1948.88206887245}
> 8 | {637.525157928467,1288.19901585579,1935.65923500061}
> 9 | {651.277820110321,1301.09531402588,1947.13559818268}
> 10 | {637.898158073425,1287.54387116432,1933.5972070694}
> 11 | {650.482151985168,1309.5760037899,1961.18471598625}
> 12 | {637.149097919464,1295.67461585999,1947.66235399246}
> 13 | {651.543440818787,1299.3868060112,1948.78301692009}
> 14 | {638.292811870575,1285.66767001152,1935.02927088737}
> 15 | {650.077238082886,1305.37611913681,1962.8950419426}
> 16 | {636.498329162598,1291.3173930645,1948.41972708702}
> 17 | {649.946799993515,1303.33863401413,1949.58992981911}
> 18 | {636.373380899429,1289.27607584,1936.36742186546}
> 19 | {647.414880990982,1297.31931090355,1945.65248298645}
> 20 | {634.09289598465,1284.170140028,1932.07456684113}
It may be possible to do the following to get the graph:
1. Have a 'time offset' like variable (initially set to zero)
2. Time the individual fit multiple function calls (in seconds I presume) (for adding the recorded times to the offset variable each time).
3. After 1 run of fit multiple, write an update like query to update `metrics_elapsed_time` by adding the total offset to every element of the array value (for the first run, an offset of 0 would be added, i.e. no offset) for the model configs trained in that round (this may be determined by the `mst_key` ranges).
4. Update the offset variable by incrementing it by the recorded time for the most recent loop run (i.e. fit multiple function call).
Let me know if anything doesn't seem clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698557841
(3)
```
PL/Python function "madlib_keras_automl"
ERROR: plpy.Error: DL: Please choose a valid distribution type for 'momentum_3': ['linear', 'log'] (plpython.c:5038)
```
Should not report `_3` or `_4` or whatever in messages since user does not know what these are. Should just report:
```
PL/Python function "madlib_keras_automl"
ERROR: plpy.Error: DL: Please choose a valid distribution type for 'momentum': ['linear', 'log'] (plpython.c:5038)
```
etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r494129949
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
+ """
+ Remove all intermediate tables created for AutoML runs/updates.
+ :param model_training: Fit Multiple function call object.
+ """
+ drop_tables([model_training.original_model_output_table, model_training.model_info_table,
+ model_training.model_summary_table, AutoMLSchema.TEMP_MST_TABLE,
+ AutoMLSchema.TEMP_MST_SUMMARY_TABLE])
+
+# @MinWarning("warning")
+class AutoMLHyperband(KerasAutoML):
+ """
+ This class implements Hyperband, an infinite-arm bandit based algorithm that speeds up random search
+ through adaptive resource allocation, successive halving (SHA) and early stopping.
+
+ This class showcases and novel hyperband implementation by executing the hyperband rounds 'diagonally'
Review comment:
I meant this:
`This class showcases and novel hyperband` -> `This class showcases a novel hyperband`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707417107
Let's be consistent with the output for hyperband and add it to the info table, even if the value will be the same for all mst keys. The table is small so that is not a problem and it may change in the future if we learn something new about how it should be presented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700952240
> (6)
> fmin definition
> https://github.com/hyperopt/hyperopt/wiki/FMin
> fmin(loss, space, algo, max_evals)
>
> Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
>
> ```
> self.num_workers = get_seg_number() * get_segments_per_host()
> ```
>
> gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
>
> (7)
> The function `get_configs_list()` may not be distributing the workload correctly to the segments.
> For
>
> ```
> num_models = 3
> num_workers = 3
> ```
>
> it returns `[(1, 3)]` which seems OK. But for
>
> ```
> num_models = 5
> num_workers = 3
> ```
>
> it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
>
> ```
> num_models = 20
> num_workers = 3
> ```
>
> it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
>
> (8)
> In `find_hyperopt_config()` confirm that the loss for _each_ model that passed to hyperopt after training, and not just the best one from the group.
>
> (9)
> defaults
> Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
@fmcquillan99
(7)
(a) As per my discussions with @reductionista , the current implementation seems to run more than 1 rounds when `number of models >= 2 * number of segments/workers in the cluster`. Sure, though the latter `[(1, 3), (4, 5)]` may have an idle worker, it may be a better approach for Hyperopt.
(b) In reference to the 2nd example, I believe this was as per our discussions to minimize number of model training function calls (and to make the schedule look relatively more 'balanced'). Observe that the former schedule runs 4 configs on 3 workers for 2 rounds/runs and continues with the 3 configs on 3 workers for all other rounds (as 3 doesn't perfectly divide 20). The latter only runs 2 configs on 3 workers at the end which also thus has an idle worker (apart from 1 more model training function call compared to the former approach).
(8) I believe all the losses are already passed on to hyperopt as part of my implementation, in the `Hyperopt TPE Update` for loop.
(9) Yes I did observe that. I tried to change the global defaults in the SQL file as per chosen automl method, but ran into errors in the process. As per my current implementation, the hyperband params are defaults and the user may need to explicitly specify hyperopt params when using hyperopt.
(10) The name changes sound good to me.
Let me know if there're more areas I could address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r495088887
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -835,10 +835,10 @@ class AutoMLHyperopt(KerasAutoML):
if param_value_list[-1] == 'linear':
return hp.uniform(cp, param_value_list[0], param_value_list[1])
elif param_value_list[-1] == 'log':
- return hp.loguniform(cp, param_value_list[0], param_value_list[1])
+ return hp.loguniform(cp, np.log(param_value_list[0]), np.log(param_value_list[1]))
else:
plpy.error("DL: Please choose a valid distribution type for '{0}': {1}".format(
- cp, ['linear', 'log']))
+ cp[:-2], ['linear', 'log']))
Review comment:
Good point, though a rare occurrence. I've made a general purpose function to deal with all of these cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-716772313
(1) install of Hyperopt component
- needs to be installed on master
- is only needed if hyperopt is called (not if only hyperband is called)
- I added a note to user docs (next commit)
(2) detailed NOTICE and CONTEXT messages can be supressed
- OK, supressed
(3) Should not report _3 or _4 or whatever in messages since user does not know what these are.
- OK, message fixed
(4) (5) hyperopt is not learning
- OK, learning now
(6) correctly set number of workers
- OK, computed correctly now
(7)
(8) confirm loss values reported to hyperopt for all configs (not best from group)
- OK, confirmed
(9) setting default values
- OK, set properly now
(10) param naming suggestions
- OK, fixed naming
(11) cumulative times
```
mst_key | metrics_elapsed_time
---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 | {3.44599199295044,5.5086510181427,7.59378290176392,9.76283192634583,11.7568578720093,14.0877909660339,16.2628719806671,18.3627188205719,20.6051309108734,22.6256680488586}
2 | {3.64783596992493,5.74869298934937,7.73203301429749,9.90109395980835,11.9493429660797,14.2302348613739,16.4116108417511,18.5028810501099,20.7483429908752,22.9229199886322}
3 | {25.3283171653748,27.2733652591705,29.2196431159973,31.2333950996399,33.013738155365,35.0193402767181,36.8298072814941,38.8312292098999,40.8666622638702,42.6906530857086}
4 | {25.4672672748566,27.4201622009277,29.3609321117401,31.3780810832977,33.1518592834473,35.1660552024841,37.1421611309052,38.971488237381,41.0169200897217,42.8317811489105}
5 | {45.613331079483,47.5778069496155,49.7638850212097,51.9869689941406,53.9853930473328,56.2250180244446,58.5185329914093,60.7617180347443,63.0875821113586,65.1909580230713}
6 | {45.7591459751129,47.7180619239807,49.9061949253082,52.1345801353455,54.1592149734497,56.3740320205688,58.6804759502411,60.9120979309082,63.2361199855804,65.3321549892426}
7 | {68.48650598526,71.0383489131927,73.7156028747559,76.0485279560089,78.6191580295563,81.2393538951874,83.7727980613708,86.2238938808441,88.8117818832397,91.40256690979}
8 | {68.6526188850403,71.2017469406128,73.8855209350586,76.213299036026,78.8485488891602,81.4080820083618,83.9574830532074,86.3879079818726,88.9709119796753,91.6372621059418}
9 | {94.9169039726257,97.4595720767975,100.022624015808,102.602681159973,104.947999954224,107.506361961365,110.27393913269,112.921897172928,115.296147108078,117.905650138855}
10 | {95.0961761474609,97.620619058609,100.199231147766,102.778658151627,105.320446014404,107.670147180557,110.440493106842,113.101521015167,115.461306095123,118.067128181458}
11 | {121.154954195023,123.238182067871,125.558053016663,127.94621014595,130.058441162109,132.456755161285,134.789571046829,136.911457061768,139.302124023438,141.447726964951}
12 | {121.302850008011,123.383195161819,125.702442169189,128.096446990967,130.208800077438,132.607058048248,134.943905115128,137.053812026978,139.451186180115,141.589867115021}
13 | {144.590404987335,146.865453958511,148.973741054535,151.335968017578,153.698374986649,155.985013008118,158.293015003204,160.394205093384,162.702248096466,164.992741107941}
14 | {144.756483078003,147.041687965393,149.141814947128,151.50594997406,153.962126970291,156.153010129929,158.461601018906,160.565253019333,162.872231960297,165.163074016571}
15 | {168.281131982803,170.96831202507,173.619775056839,176.261294126511,178.772081136703,181.430242061615,184.221279144287,186.935801029205,189.439113140106,192.158553123474}
16 | {168.455712080002,171.14265704155,173.794762134552,176.440241098404,178.940042972565,181.715918064117,184.400903940201,187.127205133438,189.61900305748,192.334141969681}
17 | {195.59405207634,197.74609208107,200.189806938171,202.618988990784,204.730000972748,207.099671125412,209.533874988556,211.708842992783,214.096724033356,216.496123075485}
18 | {195.76095199585,197.90920996666,200.347979068756,202.780756950378,204.888078927994,207.249115943909,209.746890068054,211.860005140305,214.24716591835,216.653197050095}
19 | {219.868572473526,222.539808273315,225.132962465286,227.739342451096,230.226921319962,233.084603309631,235.675887346268,238.05983042717,240.733500480652,243.349875450134}
20 | {220.287011384964,222.718561410904,225.308660268784,227.940581321716,230.405136346817,233.260821342468,235.855934381485,238.236332416534,240.908479452133,243.594557285309}
(20 rows)
```
- OK
(12) metrics_iters missing from info table
```
\d automl_output_info
Table "public.automl_output_info"
Column | Type | Modifiers
--------------------------+--------------------+-----------
mst_key | integer | not null
model_id | integer |
compile_params | text |
fit_params | text |
model_type | text |
model_size | double precision |
metrics_elapsed_time | double precision[] |
metrics_type | text[] |
loss_type | text |
training_metrics_final | double precision |
training_loss_final | double precision |
training_metrics | double precision[] |
training_loss | double precision[] |
validation_metrics_final | double precision |
validation_loss_final | double precision |
validation_metrics | double precision[] |
validation_loss | double precision[] |
metrics_iters | integer[] |
```
- OK, added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r495022842
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -835,10 +835,10 @@ class AutoMLHyperopt(KerasAutoML):
if param_value_list[-1] == 'linear':
return hp.uniform(cp, param_value_list[0], param_value_list[1])
elif param_value_list[-1] == 'log':
- return hp.loguniform(cp, param_value_list[0], param_value_list[1])
+ return hp.loguniform(cp, np.log(param_value_list[0]), np.log(param_value_list[1]))
else:
plpy.error("DL: Please choose a valid distribution type for '{0}': {1}".format(
- cp, ['linear', 'log']))
+ cp[:-2], ['linear', 'log']))
Review comment:
Would this work for `momentum_15` or `momentum_111` etc.?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-716774952
this one a bit easier to read
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
These 2 methods seems to want `hyperopt` module installed even if I don't call it. It should only error out if I select to use `hyperopt` as an autoML method.
```
%%sql
DROP TABLE IF EXISTS hb_schedule;
SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);
SELECT * FROM hb_schedule ORDER BY s DESC, i;
```
produces
```
InternalError: (psycopg2.errors.InternalError_) ImportError: No module named hyperopt (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "hyperband_schedule", line 7, in <module>
from deep_learning import madlib_keras_automl
PL/Python function "hyperband_schedule", line 21, in <module>
PL/Python function "hyperband_schedule"
[SQL: SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);]
```
Same happens with `madlib_keras_automl` when I select `hyperband` as the method I want
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699189947
I wonder if we need a `num_trials` type parameter for this method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707396551
(12)
```
\d hopt_cifar10_output_info_c100n3tperun1
Table "public.hopt_cifar10_output_info_c100n3tperun1"
Column | Type | Modifiers
--------------------------+--------------------+-----------
mst_key | integer | not null
model_id | integer |
compile_params | text |
fit_params | text |
model_type | text |
model_size | double precision |
metrics_elapsed_time | double precision[] |
metrics_type | text[] |
loss_type | text |
training_metrics_final | double precision |
training_loss_final | double precision |
training_metrics | double precision[] |
training_loss | double precision[] |
validation_metrics_final | double precision |
validation_loss_final | double precision |
validation_metrics | double precision[] |
validation_loss | double precision[] |
```
is missing from the bottom:
```
metrics_iters | integer[] |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707397232
> (12)
>
> ```
> \d hopt_cifar10_output_info_c100n3tperun1
> Table "public.hopt_cifar10_output_info_c100n3tperun1"
> Column | Type | Modifiers
> --------------------------+--------------------+-----------
> mst_key | integer | not null
> model_id | integer |
> compile_params | text |
> fit_params | text |
> model_type | text |
> model_size | double precision |
> metrics_elapsed_time | double precision[] |
> metrics_type | text[] |
> loss_type | text |
> training_metrics_final | double precision |
> training_loss_final | double precision |
> training_metrics | double precision[] |
> training_loss | double precision[] |
> validation_metrics_final | double precision |
> validation_loss_final | double precision |
> validation_metrics | double precision[] |
> validation_loss | double precision[] |
> ```
>
> is missing from the bottom:
>
> ```
> metrics_iters | integer[] |
> ```
`metrics_iters` is mentioned in the summary table itself, as its value is the same for all configs (if I recall correctly).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699189947
(6)
fmin definition
https://github.com/hyperopt/hyperopt/wiki/FMin
fmin(loss, space, algo, max_evals)
(a)
Looks like this PR is setting max_evals = num_models/num_segments. For one thing I'm not sure that
```
self.num_workers = get_seg_number() * get_segments_per_host()
```
gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2.
(b)
Looks like the logic is for each eval to find the losses for multiple models (= num segments) in parallel, then return the best loss value to hyperopt, then get a new set of params for multiple models from hyperopt. Or does it pass the loss for *each* model that is trained to hyperopt? If it is the former, then it seems like that might confuse affect hyperband in unknown ways if we are chunking/grouping like in this way.
(7)
We should re-look at the madlib hyperopt params to see if we have defined them in the right way. Do we need an option for an explicit `max_evals` ?
(8)
defaults
Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-716773016
cumulative time looks OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
(1)
These 2 methods seems to want `hyperopt` module installed even if I don't call it. It should only error out if I select to use `hyperopt` as an autoML method:
- `madlib_keras_automl`
- `hyperband_schedule`
E.g.,
```
%%sql
DROP TABLE IF EXISTS hb_schedule;
SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);
SELECT * FROM hb_schedule ORDER BY s DESC, i;
```
produces
```
InternalError: (psycopg2.errors.InternalError_) ImportError: No module named hyperopt (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "hyperband_schedule", line 7, in <module>
from deep_learning import madlib_keras_automl
PL/Python function "hyperband_schedule", line 21, in <module>
PL/Python function "hyperband_schedule"
[SQL: SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);]
```
Same happens with `madlib_keras_automl` when I select `hyperband` as the method I want
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698586366
(4)
`hyperopt` is not learning in the tests I have done so far with `rand`
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=rand',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces no-learning result with labels:
```
INFO: ***Evaluating 3 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 659.287541866 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0988999977708, loss=14.5240154266
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 663.107329845 sec
DETAIL:
Training set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
When I run `hyperband` with same models/hyperparams, it does learn:
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperband',
'R=6, eta=3, skip_last=0',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 655.647205114 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.34889999032, loss=1.83796536922
mst_key=1: metric=0.468120008707, loss=1.481554389
mst_key=2: metric=0.475380003452, loss=1.63904440403
Validation set after iteration 1:
mst_key=3: metric=0.355800002813, loss=1.83082151413
mst_key=1: metric=0.470499992371, loss=1.47972774506
mst_key=2: metric=0.470200002193, loss=1.66988003254
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707396551
(12)
```
\d hopt_cifar10_output_info_c100n3tperun1
Table "public.hopt_cifar10_output_info_c100n3tperun1"
Column | Type | Modifiers
--------------------------+--------------------+-----------
mst_key | integer | not null
model_id | integer |
compile_params | text |
fit_params | text |
model_type | text |
model_size | double precision |
metrics_elapsed_time | double precision[] |
metrics_type | text[] |
loss_type | text |
training_metrics_final | double precision |
training_loss_final | double precision |
training_metrics | double precision[] |
training_loss | double precision[] |
validation_metrics_final | double precision |
validation_loss_final | double precision |
validation_metrics | double precision[] |
validation_loss | double precision[] |
```
is missing from the bottom:
```
metrics_iters | integer[] |
```
I think that metrics_iters is *not* cumulative because that concept does not seem to make sense for hyperopt. The output table would look like:
```
mst_key | metrics_iters. | metrics_elapsed_time
---------+------------------------------------------------------
1 | {1,4,7} | {627.882262945175,1264.80915999413,1896.74943590164}
2 | {1,4,7} | {614.573877811432,1251.1022670269,1883.14707994461}
3 | {1,4,7} | {638.407788038254,1286.73397421837,1939.58357810974}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493746682
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
Do we use this `status`? I don't see it getting referenced and the dictionary already has it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493674295
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -17,27 +17,41 @@
# specific language governing permissions and limitations
# under the License.
+from ast import literal_eval
from datetime import datetime
-import plpy
+from hyperopt import hp, rand, tpe, atpe, Trials, STATUS_OK, STATUS_RUNNING
+from hyperopt.base import Domain
import math
+import numpy as np
+import plpy
from time import time
from madlib_keras_validator import MstLoaderInputValidator
-from utilities.utilities import unique_string, add_postfix, extract_keyvalue_params, \
- _assert, _assert_equal, rename_table
+from utilities.utilities import get_seg_number, get_segments_per_host, unique_string, add_postfix, \
+ extract_keyvalue_params, _assert, _assert_equal, rename_table
from utilities.control import MinWarning, SetGUC
from madlib_keras_fit_multiple_model import FitMultipleModel
from madlib_keras_model_selection import MstSearch, ModelSelectionSchema
from keras_model_arch_table import ModelArchSchema
from utilities.validate_args import table_exists, drop_tables
+import inspect
+
+def decallmethods(decorator, prefix='test_'):
Review comment:
We should use `_` in the function name. I wasn't sure if this was meant to be `de-call methods` or `dec all methods`. Also a short explanation on what this function and the inner class doing would be great.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -158,11 +174,10 @@ class HyperbandSchedule():
**locals())
plpy.execute(insert_query)
-@MinWarning("warning")
-class KerasAutoML():
- """The core AutoML function for running AutoML algorithms such as Hyperband.
- This function executes the hyperband rounds 'diagonally' to evaluate multiple configurations together
- and leverage the compute power of MPP databases such as Greenplum.
+# @MinWarning("warning")
Review comment:
Should we revert these commented lines back?
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
Review comment:
Same as before.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
Review comment:
Should be in utilities
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
Review comment:
Generic name, print_best_mst_so_far would be better.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
Review comment:
Add TODO to draw attention.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -223,10 +231,13 @@ class KerasAutoML():
{ModelArchSchema.MODEL_ARCH} JSON)
""".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
ModelArchSchema=ModelArchSchema)
- with MinWarning('warning'):
- plpy.execute(output_table_create_query)
+ # with MinWarning('warning'):
Review comment:
Same as before.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -245,40 +256,76 @@ class KerasAutoML():
validation_metrics_final DOUBLE PRECISION,
validation_loss_final DOUBLE PRECISION,
validation_metrics DOUBLE PRECISION[],
- validation_loss DOUBLE PRECISION[],
- {AutoMLSchema.METRICS_ITERS} INTEGER[])
- """.format(self=self, ModelSelectionSchema=ModelSelectionSchema,
- ModelArchSchema=ModelArchSchema, AutoMLSchema=AutoMLSchema)
- with MinWarning('warning'):
- plpy.execute(info_table_create_query)
+ validation_loss DOUBLE PRECISION[]
+ """.format(self=self,
+ ModelSelectionSchema=ModelSelectionSchema,
+ ModelArchSchema=ModelArchSchema,
+ AutoMLSchema=AutoMLSchema) + additional_cols + ")"
+ # with MinWarning('warning'):
+ plpy.execute(info_table_create_query)
- def validate_and_define_inputs(self):
+ def update_model_selection_table(self):
+ """
+ Drops and re-create the mst table to only include the best performing model configuration.
+ """
+ drop_tables([self.model_selection_table])
- if AutoMLSchema.HYPERBAND.startswith(self.automl_method.lower()):
- automl_params_dict = extract_keyvalue_params(self.automl_params,
- default_values={'R': 6, 'eta': 3, 'skip_last': 0},
- lower_case_names=False)
- # casting dict values to int
- for i in automl_params_dict:
- automl_params_dict[i] = int(automl_params_dict[i])
- _assert(len(automl_params_dict) >= 1 or len(automl_params_dict) <= 3,
- "DL: Only R, eta, and skip_last may be specified")
- for i in automl_params_dict:
- if i == AutoMLSchema.R:
- self.R = automl_params_dict[AutoMLSchema.R]
- elif i == AutoMLSchema.ETA:
- self.eta = automl_params_dict[AutoMLSchema.ETA]
- elif i == AutoMLSchema.SKIP_LAST:
- self.skip_last = automl_params_dict[AutoMLSchema.SKIP_LAST]
- else:
- plpy.error("DL: {0} is an invalid param".format(i))
- _assert(self.eta > 1, "DL: eta must be greater than 1")
- _assert(self.R >= self.eta, "DL: R should not be less than eta")
- self.s_max = int(math.floor(math.log(self.R, self.eta)))
- _assert(self.skip_last >= 0 and self.skip_last < self.s_max+1, "DL: skip_last must be " +
- "non-negative and less than {0}".format(self.s_max))
- else:
- plpy.error("DL: Only hyperband is currently supported as the automl method")
+ # only retaining best performing config
+ plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT {ModelSelectionSchema.MST_KEY}, " \
+ "{ModelSelectionSchema.MODEL_ID}, {ModelSelectionSchema.COMPILE_PARAMS}, " \
+ "{ModelSelectionSchema.FIT_PARAMS} FROM {self.model_info_table} " \
+ "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self,
+ AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema))
+
+ def generate_model_output_summary_table(self, model_training):
+ """
+ Creates and populates static values related to the AutoML workload.
+ :param model_training: Fit Multiple function call object.
+ """
+ create_query = plpy.prepare("""
+ CREATE TABLE {self.model_summary_table} AS
+ SELECT
+ $MAD${self.source_table}$MAD$::TEXT AS source_table,
+ $MAD${self.validation_table}$MAD$::TEXT AS validation_table,
+ $MAD${self.model_output_table}$MAD$::TEXT AS model,
+ $MAD${self.model_info_table}$MAD$::TEXT AS model_info,
+ (SELECT dependent_varname FROM {model_training.model_summary_table})
+ AS dependent_varname,
+ (SELECT independent_varname FROM {model_training.model_summary_table})
+ AS independent_varname,
+ $MAD${self.model_arch_table}$MAD$::TEXT AS model_arch_table,
+ $MAD${self.model_selection_table}$MAD$::TEXT AS model_selection_table,
+ $MAD${self.automl_method}$MAD$::TEXT AS automl_method,
+ $MAD${self.automl_params}$MAD$::TEXT AS automl_params,
+ $MAD${self.random_state}$MAD$::TEXT AS random_state,
+ $MAD${self.object_table}$MAD$::TEXT AS object_table,
+ {self.use_gpus} AS use_gpus,
+ (SELECT metrics_compute_frequency FROM {model_training.model_summary_table})::INTEGER
+ AS metrics_compute_frequency,
+ $MAD${self.name}$MAD$::TEXT AS name,
+ $MAD${self.description}$MAD$::TEXT AS description,
+ '{self.start_training_time}'::TIMESTAMP AS start_training_time,
+ '{self.end_training_time}'::TIMESTAMP AS end_training_time,
+ (SELECT madlib_version FROM {model_training.model_summary_table}) AS madlib_version,
+ (SELECT num_classes FROM {model_training.model_summary_table})::INTEGER AS num_classes,
+ (SELECT class_values FROM {model_training.model_summary_table}) AS class_values,
+ (SELECT dependent_vartype FROM {model_training.model_summary_table})
+ AS dependent_vartype,
+ (SELECT normalizing_const FROM {model_training.model_summary_table})
+ AS normalizing_const
+ """.format(self=self, model_training=model_training))
+
+ # with MinWarning('warning'):
+ plpy.execute(create_query)
+
+ def is_method(self, method_name):
Review comment:
`is_method` is a very generic function name, `is_automl_method` would be better.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
Review comment:
Generic name, a similar function already exists in utilities (cleanup_madlib_temp_tables, utilities.sql_in line 101).
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -404,24 +544,30 @@ class KerasAutoML():
model_id=ModelSelectionSchema.MODEL_ID,
compile_params=ModelSelectionSchema.COMPILE_PARAMS,
fit_params=ModelSelectionSchema.FIT_PARAMS)
- with MinWarning('warning'):
- plpy.execute(create_query)
+ # with MinWarning('warning'):
Review comment:
Same as before
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
+ """
+ Remove all intermediate tables created for AutoML runs/updates.
+ :param model_training: Fit Multiple function call object.
+ """
+ drop_tables([model_training.original_model_output_table, model_training.model_info_table,
+ model_training.model_summary_table, AutoMLSchema.TEMP_MST_TABLE,
+ AutoMLSchema.TEMP_MST_SUMMARY_TABLE])
+
+# @MinWarning("warning")
+class AutoMLHyperband(KerasAutoML):
+ """
+ This class implements Hyperband, an infinite-arm bandit based algorithm that speeds up random search
+ through adaptive resource allocation, successive halving (SHA) and early stopping.
+
+ This class showcases and novel hyperband implementation by executing the hyperband rounds 'diagonally'
Review comment:
Possible typo, and -> a ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493920534
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
Also there's `hyperopt_param['status']` and `hyperopt_param['result']['status']` in the API to be updated it seems.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
Also there's both `hyperopt_param['status']` and `hyperopt_param['result']['status']` in the API to be updated it seems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698539283
(2)
please turn off the detailed `NOTICE` and `CONTEXT` messages as they are showing on the console by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707400603
> I don't see it in there
>
> ```
> \d hopt_cifar10_output_summary_c100n3tperun1;
> Table "public.hopt_cifar10_output_summary_c100n3tperun1"
> Column | Type | Modifiers
> ---------------------------+-----------------------------+-----------
> source_table | text |
> validation_table | text |
> model | text |
> model_info | text |
> dependent_varname | text |
> independent_varname | text |
> model_arch_table | text |
> model_selection_table | text |
> automl_method | text |
> automl_params | text |
> random_state | text |
> object_table | text |
> use_gpus | boolean |
> metrics_compute_frequency | integer |
> name | text |
> description | text |
> start_training_time | timestamp without time zone |
> end_training_time | timestamp without time zone |
> madlib_version | text |
> num_classes | integer |
> class_values | text[] |
> dependent_vartype | text |
> normalizing_const | real |
> ```
Okay, then I believe I used the summary table format used earlier in Hyperband (instead of the fit multiple one, which includes the metrics iters).
I suppose the column can be added in the summer table generation function where its value is populated by the temp summary table from the fit multiple call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698586366
(4)
`hyperopt` is not learning in the tests I have done so far with `rand`
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=rand',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
INFO: ***Evaluating 3 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 659.287541866 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0988999977708, loss=14.5240154266
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 663.107329845 sec
DETAIL:
Training set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
When I run `hyperband` with same models/hyperparams, it does learn:
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperband',
'R=6, eta=3, skip_last=0',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 655.647205114 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.34889999032, loss=1.83796536922
mst_key=1: metric=0.468120008707, loss=1.481554389
mst_key=2: metric=0.475380003452, loss=1.63904440403
Validation set after iteration 1:
mst_key=3: metric=0.355800002813, loss=1.83082151413
mst_key=1: metric=0.470499992371, loss=1.47972774506
mst_key=2: metric=0.470200002193, loss=1.66988003254
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707396551
(12)
```
\d hopt_cifar10_output_info_c100n3tperun1
Table "public.hopt_cifar10_output_info_c100n3tperun1"
Column | Type | Modifiers
--------------------------+--------------------+-----------
mst_key | integer | not null
model_id | integer |
compile_params | text |
fit_params | text |
model_type | text |
model_size | double precision |
metrics_elapsed_time | double precision[] |
metrics_type | text[] |
loss_type | text |
training_metrics_final | double precision |
training_loss_final | double precision |
training_metrics | double precision[] |
training_loss | double precision[] |
validation_metrics_final | double precision |
validation_loss_final | double precision |
validation_metrics | double precision[] |
validation_loss | double precision[] |
```
is missing from the bottom:
```
metrics_iters | integer[] |
```
I think that metrics_iters is *not* cumulative because that concept does not seem to make sense for hyperopt. The output table would look like:
```
mst_key | metrics_iters. | metrics_elapsed_time
---------+------------------------------------------------------
1 | {1,4,7} | {627.882262945175,1264.80915999413,1896.74943590164}
2 | {1,4,7} | {614.573877811432,1251.1022670269,1883.14707994461}
3 | {1,4,7} | {638.407788038254,1286.73397421837,1939.58357810974}
```
or `{1,2,3}` or `{5,10,12}` etc. depending on metrics compute freq and number of itns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700935101
(10) param naming suggestions
change `num_models` to `num_configs` since this is what we use in `generate_model_configs()` - this is effectively the number of trials. Hyperopt calls this `max_evals`
change `num_iters` to `num_iterations` which is what we use elsewhere
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707398631
I don't see it in there
```
\d hopt_cifar10_output_summary_c100n3tperun1;
Table "public.hopt_cifar10_output_summary_c100n3tperun1"
Column | Type | Modifiers
---------------------------+-----------------------------+-----------
source_table | text |
validation_table | text |
model | text |
model_info | text |
dependent_varname | text |
independent_varname | text |
model_arch_table | text |
model_selection_table | text |
automl_method | text |
automl_params | text |
random_state | text |
object_table | text |
use_gpus | boolean |
metrics_compute_frequency | integer |
name | text |
description | text |
start_training_time | timestamp without time zone |
end_training_time | timestamp without time zone |
madlib_version | text |
num_classes | integer |
class_values | text[] |
dependent_vartype | text |
normalizing_const | real |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
khannaekta commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707411308
In the current implementation `metrics_iters` is not being populated for hyperopt.
@fmcquillan99 I can make the change to add it to the summary table as it is same for all the models or if we want to be consistent with the output for hyperband we can add it to the info table (it will be the same value for all the models though).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707399640
> mst_key | metrics_elapsed_time
> ---------+------------------------------------------------------
> 1 | {627.882262945175,1264.80915999413,1896.74943590164}
> 2 | {614.573877811432,1251.1022670269,1883.14707994461}
> 3 | {638.407788038254,1286.73397421837,1939.58357810974}
> 4 | {624.925609111786,1273.55782914162,1925.82674407959}
> 5 | {650.984225034714,1307.78145503998,1962.8148560524}
> 6 | {637.914520978928,1294.23532605171,1949.12843108177}
> 7 | {651.188719987869,1301.32990193367,1948.88206887245}
> 8 | {637.525157928467,1288.19901585579,1935.65923500061}
> 9 | {651.277820110321,1301.09531402588,1947.13559818268}
> 10 | {637.898158073425,1287.54387116432,1933.5972070694}
> 11 | {650.482151985168,1309.5760037899,1961.18471598625}
> 12 | {637.149097919464,1295.67461585999,1947.66235399246}
> 13 | {651.543440818787,1299.3868060112,1948.78301692009}
> 14 | {638.292811870575,1285.66767001152,1935.02927088737}
> 15 | {650.077238082886,1305.37611913681,1962.8950419426}
> 16 | {636.498329162598,1291.3173930645,1948.41972708702}
> 17 | {649.946799993515,1303.33863401413,1949.58992981911}
> 18 | {636.373380899429,1289.27607584,1936.36742186546}
> 19 | {647.414880990982,1297.31931090355,1945.65248298645}
> 20 | {634.09289598465,1284.170140028,1932.07456684113}
It may be possible to do the following to get the graph:
1. Have a 'time offset' like variable (initially set to zero)
2. Time the individual fit multiple function calls (in seconds I presume) (for adding the recorded times to the offset variable each time).
3. After a run of fit multiple, write an update like query to update `metrics_elapsed_time` by adding the total offset to every element of the array value (for the first run, an offset of 0 would be added, i.e. no offset) for the model configs trained in that round (this may be determined by the `mst_key` ranges).
4. Update the offset variable by incrementing it by the recorded time for the most recent loop run (i.e. fit multiple function call).
Let me know if anything doesn't seem clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707400603
> I don't see it in there
>
> ```
> \d hopt_cifar10_output_summary_c100n3tperun1;
> Table "public.hopt_cifar10_output_summary_c100n3tperun1"
> Column | Type | Modifiers
> ---------------------------+-----------------------------+-----------
> source_table | text |
> validation_table | text |
> model | text |
> model_info | text |
> dependent_varname | text |
> independent_varname | text |
> model_arch_table | text |
> model_selection_table | text |
> automl_method | text |
> automl_params | text |
> random_state | text |
> object_table | text |
> use_gpus | boolean |
> metrics_compute_frequency | integer |
> name | text |
> description | text |
> start_training_time | timestamp without time zone |
> end_training_time | timestamp without time zone |
> madlib_version | text |
> num_classes | integer |
> class_values | text[] |
> dependent_vartype | text |
> normalizing_const | real |
> ```
Okay, in that case I suppose I used the summary table format used earlier in Hyperband (instead of the fit multiple one, which includes the metrics iters).
I suppose the column can be added in the summer table generation function where its value is populated by the temp summary table from the fit multiple call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
These 2 methods seems to want `hyperopt` module installed even if I don't call it. It should only error out if I select to use `hyperopt` as an autoML method:
- `madlib_keras_automl`
- `hyperband_schedule`
```
%%sql
DROP TABLE IF EXISTS hb_schedule;
SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);
SELECT * FROM hb_schedule ORDER BY s DESC, i;
```
produces
```
InternalError: (psycopg2.errors.InternalError_) ImportError: No module named hyperopt (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "hyperband_schedule", line 7, in <module>
from deep_learning import madlib_keras_automl
PL/Python function "hyperband_schedule", line 21, in <module>
PL/Python function "hyperband_schedule"
[SQL: SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);]
```
Same happens with `madlib_keras_automl` when I select `hyperband` as the method I want
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
(1)
These 2 methods seems to want `hyperopt` module installed even if I don't call it. It should only error out if I select to use `hyperopt` as an autoML method:
- `madlib_keras_automl`
- `hyperband_schedule`
E.g.,
```
%%sql
DROP TABLE IF EXISTS hb_schedule;
SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);
SELECT * FROM hb_schedule ORDER BY s DESC, i;
```
produces
```
InternalError: (psycopg2.errors.InternalError_) ImportError: No module named hyperopt (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "hyperband_schedule", line 7, in <module>
from deep_learning import madlib_keras_automl
PL/Python function "hyperband_schedule", line 21, in <module>
PL/Python function "hyperband_schedule"
[SQL: SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);]
```
Same happens with `madlib_keras_automl` when I select `hyperband` as the method I want. So we need to figure out how hyperopt is installed with madlib.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta merged pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
khannaekta merged pull request #518:
URL: https://github.com/apache/madlib/pull/518
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] khannaekta commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
khannaekta commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707404066
@Advitya17 Thanks for looking into this! Yes, that is one of the ways we can do to fix the times. I will go ahead and make the necessary changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698612856
output tables
```
madlib=# select * from automl_cifar10_output_summary;
-[ RECORD 1 ]-------------+-----------------------------------------
source_table | cifar10_train_packed
validation_table | cifar10_val_packed
model | automl_cifar10_output
model_info | automl_cifar10_output_info
dependent_varname | y
independent_varname | x
model_arch_table | model_arch_library
model_selection_table | automl_cifar10_mst_table
automl_method | hyperopt
automl_params | num_models=4, num_iters=2, algorithm=tpe
random_state | None
object_table | None
use_gpus | f
metrics_compute_frequency | 1
name | None
description | None
start_training_time | 2020-09-24 21:27:43
end_training_time | 2020-09-24 21:57:21
madlib_version | 1.18.0-dev
num_classes | 10
class_values | {0,1,2,3,4,5,6,7,8,9}
dependent_vartype | text
normalizing_const | 256
```
and
```
madlib=# select * from automl_cifar10_output_info;
-[ RECORD 1 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 1
model_id | 1
compile_params | optimizer='SGD(lr=1.0060862144391352,momentum=2.477852193264555)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=128
model_type | madlib_keras
model_size | 4886.20019531
metrics_elapsed_time | {854.796041965485,1740.91218686104}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
-[ RECORD 2 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 3
model_id | 2
compile_params | optimizer='SGD(lr=1.0019632479006402,momentum=2.468115689403145)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=128
model_type | madlib_keras
model_size | 2159.70019531
metrics_elapsed_time | {867.580676794052,1753.89660692215}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
-[ RECORD 3 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 4
model_id | 2
compile_params | optimizer='Adam(lr=1.009549897036113)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=64
model_type | madlib_keras
model_size | 2159.70019531
metrics_elapsed_time | {880.187488794327,1766.73845982552}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.0999400019645691
training_loss_final | 14.5072526931763
training_metrics | {0.0996000021696091,0.0999400019645691}
training_loss | {14.5127325057983,14.5072526931763}
validation_metrics_final | 0.100100003182888
validation_loss_final | 14.5046730041504
validation_metrics | {0.101099997758865,0.100100003182888}
validation_loss | {14.4885559082031,14.5046730041504}
-[ RECORD 4 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | optimizer='SGD(lr=1.0039632314582876,momentum=2.466481495072043)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=64
model_type | madlib_keras
model_size | 4886.20019531
metrics_elapsed_time | {885.122605800629,1772.07805681229}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698612856
output tables
```
madlib=# select * from automl_cifar10_output_summary;
-[ RECORD 1 ]-------------+-----------------------------------------
source_table | cifar10_train_packed
validation_table | cifar10_val_packed
model | automl_cifar10_output
model_info | automl_cifar10_output_info
dependent_varname | y
independent_varname | x
model_arch_table | model_arch_library
model_selection_table | automl_cifar10_mst_table
automl_method | hyperopt
automl_params | num_models=4, num_iters=2, algorithm=tpe
random_state | None
object_table | None
use_gpus | f
metrics_compute_frequency | 1
name | None
description | None
start_training_time | 2020-09-24 21:27:43
end_training_time | 2020-09-24 21:57:21
madlib_version | 1.18.0-dev
num_classes | 10
class_values | {0,1,2,3,4,5,6,7,8,9}
dependent_vartype | text
normalizing_const | 256
```
and
```
madlib=# select * from automl_cifar10_output_info;
-[ RECORD 1 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 1
model_id | 1
compile_params | optimizer='SGD(lr=1.0060862144391352,momentum=2.477852193264555)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=128
model_type | madlib_keras
model_size | 4886.20019531
metrics_elapsed_time | {854.796041965485,1740.91218686104}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
-[ RECORD 2 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 3
model_id | 2
compile_params | optimizer='SGD(lr=1.0019632479006402,momentum=2.468115689403145)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=128
model_type | madlib_keras
model_size | 2159.70019531
metrics_elapsed_time | {867.580676794052,1753.89660692215}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
-[ RECORD 3 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 4
model_id | 2
compile_params | optimizer='Adam(lr=1.009549897036113)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=64
model_type | madlib_keras
model_size | 2159.70019531
metrics_elapsed_time | {880.187488794327,1766.73845982552}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.0999400019645691
training_loss_final | 14.5072526931763
training_metrics | {0.0996000021696091,0.0999400019645691}
training_loss | {14.5127325057983,14.5072526931763}
validation_metrics_final | 0.100100003182888
validation_loss_final | 14.5046730041504
validation_metrics | {0.101099997758865,0.100100003182888}
validation_loss | {14.4885559082031,14.5046730041504}
-[ RECORD 4 ]------------+-----------------------------------------------------------------------------------------------------------------------
mst_key | 2
model_id | 1
compile_params | optimizer='SGD(lr=1.0039632314582876,momentum=2.466481495072043)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=64
model_type | madlib_keras
model_size | 4886.20019531
metrics_elapsed_time | {885.122605800629,1772.07805681229}
metrics_type | {accuracy}
loss_type | categorical_crossentropy
training_metrics_final | 0.100000001490116
training_loss_final | NaN
training_metrics | {0.100000001490116,0.100000001490116}
training_loss | {NaN,NaN}
validation_metrics_final | 0.100000001490116
validation_loss_final | NaN
validation_metrics | {0.100000001490116,0.100000001490116}
validation_loss | {NaN,NaN}
```
```
madlib=# select * from automl_cifar10_mst_table;
-[ RECORD 1 ]--+--------------------------------------------------------------------------------------------
mst_key | 4
model_id | 2
compile_params | optimizer='Adam(lr=1.009549897036113)',metrics=['accuracy'],loss='categorical_crossentropy'
fit_params | epochs=1,batch_size=64
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699189947
(6)
fmin definition
https://github.com/hyperopt/hyperopt/wiki/FMin
fmin(loss, space, algo, max_evals)
Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
```
self.num_workers = get_seg_number() * get_segments_per_host()
```
gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
(7)
The function `get_configs_list()` may not be distributing the workload correctly to the segments.
For
```
num_models = 3
num_workers = 3
```
it returns `[(1, 3)]` which seems OK. But for
```
num_models = 5
num_workers = 3
```
it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
```
num_models = 20
num_workers = 3
```
it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
(8)
In `find_hyperopt_config()` confirm that the loss for *each* model that passed to hyperopt after training, and not just the best one from the group.
(9)
defaults
Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-716773016
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700952240
> (6)
> fmin definition
> https://github.com/hyperopt/hyperopt/wiki/FMin
> fmin(loss, space, algo, max_evals)
>
> Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
>
> ```
> self.num_workers = get_seg_number() * get_segments_per_host()
> ```
>
> gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
>
> (7)
> The function `get_configs_list()` may not be distributing the workload correctly to the segments.
> For
>
> ```
> num_models = 3
> num_workers = 3
> ```
>
> it returns `[(1, 3)]` which seems OK. But for
>
> ```
> num_models = 5
> num_workers = 3
> ```
>
> it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
>
> ```
> num_models = 20
> num_workers = 3
> ```
>
> it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
>
> (8)
> In `find_hyperopt_config()` confirm that the loss for _each_ model that passed to hyperopt after training, and not just the best one from the group.
>
> (9)
> defaults
> Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
@fmcquillan99
(7)
(a) As per my discussions with @reductionista , the current implementation seems to run more than 1 rounds when `number of models >= 2 * number of segments/workers in the cluster`. Sure, though the latter `[(1, 3), (4, 5)]` may have an idle worker, it may be a better approach for Hyperopt.
(b) In reference to the 2nd example, I believe this was as per our discussions to minimize number of model training function calls (and to make the schedule look relatively more 'balanced'). Observe that the former schedule runs 4 configs on 3 workers for 2 rounds/runs and continues with the 3 configs on 3 workers for all other rounds (as 3 doesn't perfectly divide 20). The latter runs 2 configs on 3 workers at the end which also thus has an idle worker (apart from 1 more model training function call compared to the former approach).
(8) I believe all the losses are already passed on to hyperopt as part of my implementation, in the `Hyperopt TPE Update` for loop.
(9) Yes I did observe that. I tried to change the global defaults in the SQL file as per chosen automl method, but ran into errors in the process. As per my current implementation, the hyperband params are defaults and the user may need to explicitly specify hyperopt params when using hyperopt.
(10) The name changes sound good to me.
Let me know if there're more areas I could address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493918825
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
Yeah it seems it's apparently used internally by the hyperopt module during `refresh` and to explore further configs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493918484
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -17,27 +17,41 @@
# specific language governing permissions and limitations
# under the License.
+from ast import literal_eval
from datetime import datetime
-import plpy
+from hyperopt import hp, rand, tpe, atpe, Trials, STATUS_OK, STATUS_RUNNING
+from hyperopt.base import Domain
import math
+import numpy as np
+import plpy
from time import time
from madlib_keras_validator import MstLoaderInputValidator
-from utilities.utilities import unique_string, add_postfix, extract_keyvalue_params, \
- _assert, _assert_equal, rename_table
+from utilities.utilities import get_seg_number, get_segments_per_host, unique_string, add_postfix, \
+ extract_keyvalue_params, _assert, _assert_equal, rename_table
from utilities.control import MinWarning, SetGUC
from madlib_keras_fit_multiple_model import FitMultipleModel
from madlib_keras_model_selection import MstSearch, ModelSelectionSchema
from keras_model_arch_table import ModelArchSchema
from utilities.validate_args import table_exists, drop_tables
+import inspect
+
+def decallmethods(decorator, prefix='test_'):
Review comment:
I inserted this to make the class inheritance work earlier. On further inspection, it seems I may not need this function.
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -158,11 +174,10 @@ class HyperbandSchedule():
**locals())
plpy.execute(insert_query)
-@MinWarning("warning")
-class KerasAutoML():
- """The core AutoML function for running AutoML algorithms such as Hyperband.
- This function executes the hyperband rounds 'diagonally' to evaluate multiple configurations together
- and leverage the compute power of MPP databases such as Greenplum.
+# @MinWarning("warning")
Review comment:
They error out if I do that:
```
ERROR: TypeError: Error when calling the metaclass bases
function() argument 1 must be code, not str (plpy_elog.c:121)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r494130605
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
OK we should leave a short comment to make sure it doesn't get refactored out in the future
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r495189361
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
Review comment:
I reverted back to my function as I was unfortunately facing pipeline issues with the utility function. I've still commented out the related lines in case anyone wants to experiment with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707400603
> I don't see it in there
>
> ```
> \d hopt_cifar10_output_summary_c100n3tperun1;
> Table "public.hopt_cifar10_output_summary_c100n3tperun1"
> Column | Type | Modifiers
> ---------------------------+-----------------------------+-----------
> source_table | text |
> validation_table | text |
> model | text |
> model_info | text |
> dependent_varname | text |
> independent_varname | text |
> model_arch_table | text |
> model_selection_table | text |
> automl_method | text |
> automl_params | text |
> random_state | text |
> object_table | text |
> use_gpus | boolean |
> metrics_compute_frequency | integer |
> name | text |
> description | text |
> start_training_time | timestamp without time zone |
> end_training_time | timestamp without time zone |
> madlib_version | text |
> num_classes | integer |
> class_values | text[] |
> dependent_vartype | text |
> normalizing_const | real |
> ```
Okay, in that case I suppose I used the summary table format used earlier in Hyperband (instead of the fit multiple one, which includes the metrics iters).
I suppose the column can be added in the summary table generation function where its value is populated by the temp summary table from the fit multiple call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700952240
> (6)
> fmin definition
> https://github.com/hyperopt/hyperopt/wiki/FMin
> fmin(loss, space, algo, max_evals)
>
> Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
>
> ```
> self.num_workers = get_seg_number() * get_segments_per_host()
> ```
>
> gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
>
> (7)
> The function `get_configs_list()` may not be distributing the workload correctly to the segments.
> For
>
> ```
> num_models = 3
> num_workers = 3
> ```
>
> it returns `[(1, 3)]` which seems OK. But for
>
> ```
> num_models = 5
> num_workers = 3
> ```
>
> it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
>
> ```
> num_models = 20
> num_workers = 3
> ```
>
> it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
>
> (8)
> In `find_hyperopt_config()` confirm that the loss for _each_ model that passed to hyperopt after training, and not just the best one from the group.
>
> (9)
> defaults
> Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
@fmcquillan99
(7)
(a) As per my discussions with @reductionista , the current implementation seems to run more than 1 rounds when `number of models >= 2 * number of segments/workers in the cluster`. Sure, though the latter `[(1, 3), (4, 5)]` may have an idle worker, it may be a better approach for Hyperopt.
(b) In reference to the 2nd example, I believe this was as per our discussions to minimize number of model training function calls (and to make the schedule look relatively more 'balanced'). Observe that the former schedule runs 4 configs on 3 workers for 2 rounds/runs and continues with the 3 configs on 3 workers for all other rounds (as 3 doesn't perfectly divide 20). The latter runs 2 configs on 3 workers at the end which also thus has an idle worker (apart from 1 more model training function call compared to the former approach).
(8) I believe all the losses are already passed on to hyperopt as part of my implementation, in the `Hyperopt TPE Update` for loop (starting line 753 of the AutoML python file).
(9) Yes I did observe that. I tried to change the global defaults in the SQL file as per chosen automl method, but ran into errors in the process. As per my current implementation, the hyperband params are defaults and the user may need to explicitly specify hyperopt params when using hyperopt.
(10) The name changes sound good to me.
Let me know if there're more areas I could address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r494129949
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
+ """
+ Remove all intermediate tables created for AutoML runs/updates.
+ :param model_training: Fit Multiple function call object.
+ """
+ drop_tables([model_training.original_model_output_table, model_training.model_info_table,
+ model_training.model_summary_table, AutoMLSchema.TEMP_MST_TABLE,
+ AutoMLSchema.TEMP_MST_SUMMARY_TABLE])
+
+# @MinWarning("warning")
+class AutoMLHyperband(KerasAutoML):
+ """
+ This class implements Hyperband, an infinite-arm bandit based algorithm that speeds up random search
+ through adaptive resource allocation, successive halving (SHA) and early stopping.
+
+ This class showcases and novel hyperband implementation by executing the hyperband rounds 'diagonally'
Review comment:
I meant this:
`This class showcases and novel hyperband` -> `This class showcases a novel hyperband`
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
OK we should leave a short comment to make sure it doesn't get refactored out in the future
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707397232
> (12)
>
> ```
> \d hopt_cifar10_output_info_c100n3tperun1
> Table "public.hopt_cifar10_output_info_c100n3tperun1"
> Column | Type | Modifiers
> --------------------------+--------------------+-----------
> mst_key | integer | not null
> model_id | integer |
> compile_params | text |
> fit_params | text |
> model_type | text |
> model_size | double precision |
> metrics_elapsed_time | double precision[] |
> metrics_type | text[] |
> loss_type | text |
> training_metrics_final | double precision |
> training_loss_final | double precision |
> training_metrics | double precision[] |
> training_loss | double precision[] |
> validation_metrics_final | double precision |
> validation_loss_final | double precision |
> validation_metrics | double precision[] |
> validation_loss | double precision[] |
> ```
>
> is missing from the bottom:
>
> ```
> metrics_iters | integer[] |
> ```
`metrics_iters` should be mentioned in the summary table itself, as its value is the same for all configs (if I recall correctly).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700952240
> (6)
> fmin definition
> https://github.com/hyperopt/hyperopt/wiki/FMin
> fmin(loss, space, algo, max_evals)
>
> Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
>
> ```
> self.num_workers = get_seg_number() * get_segments_per_host()
> ```
>
> gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
>
> (7)
> The function `get_configs_list()` may not be distributing the workload correctly to the segments.
> For
>
> ```
> num_models = 3
> num_workers = 3
> ```
>
> it returns `[(1, 3)]` which seems OK. But for
>
> ```
> num_models = 5
> num_workers = 3
> ```
>
> it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
>
> ```
> num_models = 20
> num_workers = 3
> ```
>
> it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
>
> (8)
> In `find_hyperopt_config()` confirm that the loss for _each_ model that passed to hyperopt after training, and not just the best one from the group.
>
> (9)
> defaults
> Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
@fmcquillan99
(7)
(a) As per my discussions with @reductionista , the current implementation seems to run more than 1 rounds when `number of models >= 2 * number of segments/workers in the cluster`. Sure, though the latter `[(1, 3), (4, 5)]` may have an idle worker, it may be a better approach for Hyperopt.
(b) In reference to the 2nd example, I believe this was as per our discussions to minimize number of model training function calls. Observe that the former schedule runs 4 configs on 3 workers for 2 rounds/runs and continues with the 3 configs on 3 workers for all other rounds (as 3 doesn't perfectly divide 20). The latter only runs 2 configs on 3 workers at the end which also thus has an idle worker (apart from 1 more model training function call compared to the former approach).
(8) I believe all the losses are already passed on to hyperopt as part of my implementation, in the `Hyperopt TPE Update` for loop.
(9) Yes I did observe that. I tried to change the global defaults in the SQL file as per chosen automl method, but ran into errors in the process. As per my current implementation, the hyperband params are defaults and the user may need to explicitly specify hyperopt params when using hyperopt.
(10) The name changes sound good to me.
Let me know if there're more areas I could address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707396018
(11)
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[2],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [128], 'epochs': [1]}$$,
'hyperopt',
'num_configs=100, num_iterations=3, algorithm=tpe',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
select mst_key, metrics_elapsed_time from hopt_cifar10_output_info_c100n3tperun1 order by mst_key;
mst_key | metrics_elapsed_time
---------+------------------------------------------------------
1 | {627.882262945175,1264.80915999413,1896.74943590164}
2 | {614.573877811432,1251.1022670269,1883.14707994461}
3 | {638.407788038254,1286.73397421837,1939.58357810974}
4 | {624.925609111786,1273.55782914162,1925.82674407959}
5 | {650.984225034714,1307.78145503998,1962.8148560524}
6 | {637.914520978928,1294.23532605171,1949.12843108177}
7 | {651.188719987869,1301.32990193367,1948.88206887245}
8 | {637.525157928467,1288.19901585579,1935.65923500061}
9 | {651.277820110321,1301.09531402588,1947.13559818268}
10 | {637.898158073425,1287.54387116432,1933.5972070694}
11 | {650.482151985168,1309.5760037899,1961.18471598625}
12 | {637.149097919464,1295.67461585999,1947.66235399246}
13 | {651.543440818787,1299.3868060112,1948.78301692009}
14 | {638.292811870575,1285.66767001152,1935.02927088737}
15 | {650.077238082886,1305.37611913681,1962.8950419426}
16 | {636.498329162598,1291.3173930645,1948.41972708702}
17 | {649.946799993515,1303.33863401413,1949.58992981911}
18 | {636.373380899429,1289.27607584,1936.36742186546}
19 | {647.414880990982,1297.31931090355,1945.65248298645}
20 | {634.09289598465,1284.170140028,1932.07456684113}
21 | {684.188683986664,1330.76613283157,1980.06952404976}
22 | {670.57110285759,1317.26948404312,1966.71574902534}
23 | {651.919174909592,1304.4483089447,1949.68165898323}
24 | {637.834177970886,1290.82985186577,1936.79075098038}
25 | {655.077679872513,1298.5494530201,1945.88967299461}
26 | {642.029677867889,1285.51564288139,1932.42474102974}
27 | {647.519994020462,1291.86638307571,1940.63740706444}
28 | {634.281761884689,1277.6536450386,1927.27670001984}
29 | {649.227880954742,1295.78293895721,1939.8649418354}
30 | {635.665500879288,1282.13676691055,1926.41391181946}
31 | {646.891736984253,1292.39306688309,1939.45323801041}
32 | {633.289309024811,1278.57114887238,1925.29498195648}
33 | {645.044659852982,1294.90844988823,1934.83831381798}
34 | {631.370570898056,1281.09457278252,1921.84612298012}
35 | {641.974558115005,1287.15608096123,1933.71887016296}
36 | {628.698891162872,1273.55222702026,1920.55405902863}
37 | {645.359504938126,1286.81019496918,1936.24443602562}
38 | {632.138350963593,1273.69606280327,1921.84107279778}
39 | {645.965959072113,1291.83859109879,1934.28793001175}
40 | {631.842108011246,1278.0611550808,1921.37190818787}
41 | {644.744366168976,1284.78299307823,1926.56299805641}
42 | {631.000986099243,1271.36041498184,1912.94822907448}
43 | {646.629382133484,1289.81746912003,1938.30674505234}
44 | {633.08934211731,1276.87801408768,1924.48914599419}
45 | {645.794209003448,1289.52689504623,1928.10301995277}
46 | {632.544068098068,1276.46976709366,1915.18053913116}
47 | {648.967436075211,1296.18764710426,1942.72390508652}
48 | {635.180493116379,1282.65127015114,1928.7596950531}
49 | {642.685960054398,1281.86446094513,1924.27530503273}
50 | {628.867849111557,1268.81643414497,1910.81892299652}
51 | {642.25784611702,1285.31040906906,1931.93548893929}
52 | {628.323985099792,1271.58122897148,1918.26403403282}
53 | {641.810137033463,1285.25560903549,1929.13687109947}
54 | {628.287924051285,1271.25157594681,1915.76375198364}
55 | {645.721904993057,1285.12515211105,1928.89319014549}
56 | {632.100169181824,1271.49339413643,1915.78533101082}
57 | {639.617475986481,1275.61342597008,1915.39825177193}
58 | {626.503052949905,1262.02722096443,1901.32048177719}
59 | {635.73348903656,1280.25348901749,1919.08957004547}
60 | {622.708857059479,1266.27361917496,1905.91475009918}
61 | {640.503421068192,1279.74774312973,1920.75052404404}
62 | {627.23721909523,1266.51659917831,1906.79211616516}
63 | {642.081022977829,1277.49766278267,1916.42965888977}
64 | {628.109483957291,1264.7400097847,1903.20749592781}
65 | {640.490365028381,1283.00291013718,1924.50952506065}
66 | {627.058392047882,1269.47951006889,1910.89861011505}
67 | {636.89220905304,1277.09766602516,1910.96581101418}
68 | {624.116094112396,1263.90056610107,1898.12311601639}
69 | {635.469167947769,1269.55181598663,1904.01561403275}
70 | {622.43970489502,1256.59510493279,1890.07668590546}
71 | {636.119792938232,1276.15144109726,1913.03098011017}
72 | {622.38174700737,1262.04434704781,1899.77307701111}
73 | {635.743231058121,1276.63025021553,1917.14123606682}
74 | {622.766848087311,1263.39644908905,1903.07419419289}
75 | {630.598103046417,1267.86923599243,1899.72781896591}
76 | {617.88782787323,1254.63754701614,1886.25667786598}
77 | {632.587329149246,1270.71484613419,1902.50029301643}
78 | {619.189999103546,1257.35159301758,1889.55543804169}
79 | {631.962325811386,1268.57928681374,1901.31368494034}
80 | {618.959751844406,1255.32190680504,1888.31697797775}
81 | {634.275179862976,1261.66588997841,1895.5607509613}
82 | {619.90931391716,1248.77173399925,1882.48632383347}
83 | {633.14914894104,1265.85549592972,1897.17652893066}
84 | {619.785959005356,1252.29544901848,1884.09737896919}
85 | {627.898339033127,1261.44491815567,1893.74317097664}
86 | {615.068039178848,1248.09124517441,1879.57727813721}
87 | {633.1245470047,1268.63815689087,1904.82987594604}
88 | {619.971349954605,1255.54224300385,1891.63547801971}
89 | {634.308418035507,1267.44264101982,1903.04254603386}
90 | {621.244497060776,1254.47942519188,1889.07282710075}
91 | {638.813256025314,1286.2553050518,1923.13115119934}
92 | {624.587430000305,1271.47850799561,1909.81140208244}
93 | {635.323696136475,1271.31673598289,1910.10420203209}
94 | {622.143538951874,1258.46868395805,1896.75948596001}
95 | {636.06889295578,1269.57318091393,1905.99937295914}
96 | {621.923580884933,1255.54979085922,1892.35493183136}
97 | {635.867208957672,1275.91143798828,1907.85138106346}
98 | {622.153614997864,1262.45370602608,1895.00853896141}
99 | {632.058423995972,1264.59007382393,1902.03439188004}
100 | {618.850169897079,1251.59025883675,1889.0108628273}
(100 rows)
```
These times should be cumulative, not relative to when the trial started. We want to be able to do a plot that looks like:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699189947
(6)
fmin definition
https://github.com/hyperopt/hyperopt/wiki/FMin
fmin(loss, space, algo, max_evals)
Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
```
self.num_workers = get_seg_number() * get_segments_per_host()
```
gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
(7)
The function `get_configs_list()` may not be distributing the workload correctly to the segments.
For
```
num_models = 3
num_workers = 3
```
it returns `[(1, 3)]` which seems OK. But for
```
num_models = 5
num_workers = 3
```
it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
```
num_models = 20
num_workers = 3
```
it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
(8)
In `find_hyperopt_config()` confirm that the loss for *each* model that passed to hyperopt after training, and not just the best one from the group.
(9)
defaults
Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error. Should there be defaults for hyperband and hyperopt, or should they be mandatory?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699189947
(6)
fmin definition
https://github.com/hyperopt/hyperopt/wiki/FMin
fmin(loss, space, algo, max_evals)
(a)
Looks like this PR is setting max_evals = num_models/num_segments. For one thing I'm not sure that
```
self.num_workers = get_seg_number() * get_segments_per_host()
```
gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
(b)
Looks like the logic is for each eval to find the losses for multiple models (= num segments) in parallel, then return the best loss value to hyperopt, then get a new set of params for multiple models from hyperopt. Or does it pass the loss for *each* model that is trained to hyperopt? If it is the former, then it seems like that might confuse affect hyperband in unknown ways if we are chunking/grouping like in this way.
(7)
We should re-look at the madlib hyperopt params to see if we have defined them in the right way. Do we need an option for an explicit `max_evals` ?
(8)
defaults
Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-716907845
(100) and (101)
- error messages look OK now
LGTM after merge conflicts fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493918605
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -223,10 +231,13 @@ class KerasAutoML():
{ModelArchSchema.MODEL_ARCH} JSON)
""".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
ModelArchSchema=ModelArchSchema)
- with MinWarning('warning'):
- plpy.execute(output_table_create_query)
+ # with MinWarning('warning'):
Review comment:
See above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] orhankislal commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
orhankislal commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493747805
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
+ hyperopt_param['result'] = {'loss': loss_val, 'status': STATUS_OK}
+ trials.refresh()
+
+ # stacks info of all model configs together
+ self.update_model_output_and_info_tables(model_training)
+
+ self.print_best_so_far()
+
+ self.end_training_time = self.get_current_timestamp()
+ self.update_model_selection_table()
+ self.generate_model_output_summary_table(model_training)
+ self.remove_temp_tables(model_training)
+
+ def get_configs_list(self):
"""
- Creates and populates static values related to the AutoML workload.
- :param model_training: Fit Multiple function call object.
+ Gets schedule to evaluate model configs
+ :return: Model configs evaluation schedule
"""
- create_query = plpy.prepare("""
- CREATE TABLE {self.model_summary_table} AS
- SELECT
- $MAD${self.source_table}$MAD$::TEXT AS source_table,
- $MAD${self.validation_table}$MAD$::TEXT AS validation_table,
- $MAD${self.model_output_table}$MAD$::TEXT AS model,
- $MAD${self.model_info_table}$MAD$::TEXT AS model_info,
- (SELECT dependent_varname FROM {model_training.model_summary_table})
- AS dependent_varname,
- (SELECT independent_varname FROM {model_training.model_summary_table})
- AS independent_varname,
- $MAD${self.model_arch_table}$MAD$::TEXT AS model_arch_table,
- $MAD${self.model_selection_table}$MAD$::TEXT AS model_selection_table,
- $MAD${self.automl_method}$MAD$::TEXT AS automl_method,
- $MAD${self.automl_params}$MAD$::TEXT AS automl_params,
- $MAD${self.random_state}$MAD$::TEXT AS random_state,
- $MAD${self.object_table}$MAD$::TEXT AS object_table,
- {self.use_gpus} AS use_gpus,
- (SELECT metrics_compute_frequency FROM {model_training.model_summary_table})::INTEGER
- AS metrics_compute_frequency,
- $MAD${self.name}$MAD$::TEXT AS name,
- $MAD${self.description}$MAD$::TEXT AS description,
- '{self.start_training_time}'::TIMESTAMP AS start_training_time,
- '{self.end_training_time}'::TIMESTAMP AS end_training_time,
- (SELECT madlib_version FROM {model_training.model_summary_table}) AS madlib_version,
- (SELECT num_classes FROM {model_training.model_summary_table})::INTEGER AS num_classes,
- (SELECT class_values FROM {model_training.model_summary_table}) AS class_values,
- (SELECT dependent_vartype FROM {model_training.model_summary_table})
- AS dependent_vartype,
- (SELECT normalizing_const FROM {model_training.model_summary_table})
- AS normalizing_const
- """.format(self=self, model_training=model_training))
+ prop = self.num_models // self.num_workers
+ if prop == 0:
+ return [(1, self.num_models)]
+ lst = [self.num_workers for _ in range(prop)]
- with MinWarning('warning'):
- plpy.execute(create_query)
+ # balancing remaining models in initial rounds
+ for i in range(self.num_models - prop * self.num_workers):
+ lst[i%len(lst)] += 1
- def remove_temp_tables(self, model_training):
+ return [(sum(lst[:e])+1, sum(lst[:e])+lst[e]) if e != 0 else (1, lst[e]) for e in range(len(lst))]
+
+ def get_search_space(self):
"""
- Remove all intermediate tables created for AutoML runs/updates.
- :param model_training: Fit Multiple function call object.
+ Converts user inputs to hyperopt search space.
+ :return: Hyperopt search space
"""
- drop_tables([model_training.original_model_output_table, model_training.model_info_table,
- model_training.model_summary_table, AutoMLSchema.TEMP_MST_TABLE,
- AutoMLSchema.TEMP_MST_SUMMARY_TABLE])
+
+ # initial params (outside 'optimizer_params_list')
+ hyperopt_search_dict = {}
+ hyperopt_search_dict['model_id'] = self.get_hyperopt_exps('model_id', self.model_id_list)
+
+
+ for j in self.fit_params_grid:
+ hyperopt_search_dict[j] = self.get_hyperopt_exps(j, self.fit_params_grid[j])
+
+ for i in self.compile_params_grid:
+ if i == ModelSelectionSchema.OPTIMIZER_PARAMS_LIST:
Review comment:
Why not use the opposite of the condition to get rid of continue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-700952240
> (6)
> fmin definition
> https://github.com/hyperopt/hyperopt/wiki/FMin
> fmin(loss, space, algo, max_evals)
>
> Looks like this PR is setting max_evals = num_models/num_segments in `get_configs_list()`. For one thing I'm not sure that
>
> ```
> self.num_workers = get_seg_number() * get_segments_per_host()
> ```
>
> gives total number of workers? On a 1 host, 2 segments-per-host database this returned 4 instead of the expected 2. Also this needs to be consistent with the distribution rules set in the mini-batch preprocessor.
>
> (7)
> The function `get_configs_list()` may not be distributing the workload correctly to the segments.
> For
>
> ```
> num_models = 3
> num_workers = 3
> ```
>
> it returns `[(1, 3)]` which seems OK. But for
>
> ```
> num_models = 5
> num_workers = 3
> ```
>
> it returns `[(1, 5)]` which does not seem OK. It will not do any hyperopt updates if all 5 configs are grouped together. I would have expected `[(1, 3), (4, 5)]`. For
>
> ```
> num_models = 20
> num_workers = 3
> ```
>
> it returns `[(1, 4), (5, 8), (9, 11), (12, 14), (15, 17), (18, 20)]` which means it is running 4 configs on 3 segments multiple times which does not seem efficient. I would have expected something like `[(1, 3), (4, 6), (7, 9), (10, 12), (13, 15), (16, 18),(19, 20)]` which runs 3 configs at a time on the 3 segments.
>
> (8)
> In `find_hyperopt_config()` confirm that the loss for _each_ model that passed to hyperopt after training, and not just the best one from the group.
>
> (9)
> defaults
> Seems like hyperband defaults are being used for hyperopt in the case that use does not specify hyperband is not specified. That will probably throw an error
(7)
(a) As per my discussions with Domino, the current implementation seems to run more than 1 rounds when `number of models >= 2 * number of segments/workers in the cluster`. Sure, though the latter `[(1, 3), (4, 5)]` may have an idle worker, it may be a better approach for Hyperopt.
(b) In reference to the 2nd example, I believe this was as per our discussions to minimize number of model training function calls. Observe that the former schedule runs 4 configs on 3 workers for 2 rounds/runs and continues with the 3 configs on 3 workers for all other rounds (as 3 doesn't perfectly divide 20). The latter only runs 2 configs on 3 workers at the end which also thus has an idle worker (apart from 1 more model training function call compared to the former approach).
(8) I believe all the losses are already passed on to hyperopt as part of my implementation, in the `Hyperopt TPE Update` for loop.
(9) Yes I did observe that. I tried to change the global defaults in the SQL file as per chosen automl method, but ran into errors in the process. As per my current implementation, the hyperband params are defaults and the user may need to explicitly specify hyperopt params when using hyperopt.
(10) The name changes sound good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698586366
(4)
`hyperopt` is not learning with cifar10 in the tests I have done so far with `rand`
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=rand',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces no-learning result with labels:
```
INFO: ***Evaluating 3 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 659.287541866 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0988999977708, loss=14.5240154266
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 663.107329845 sec
DETAIL:
Training set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
When I run `hyperband` with same models/hyperparams, it does learn:
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperband',
'R=6, eta=3, skip_last=0',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 655.647205114 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.34889999032, loss=1.83796536922
mst_key=1: metric=0.468120008707, loss=1.481554389
mst_key=2: metric=0.475380003452, loss=1.63904440403
Validation set after iteration 1:
mst_key=3: metric=0.355800002813, loss=1.83082151413
mst_key=1: metric=0.470499992371, loss=1.47972774506
mst_key=2: metric=0.470200002193, loss=1.66988003254
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
Also `hyperopt` with `tpe` is not learning cifar10
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=tpe',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 648.608314991 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0999400019646, loss=14.5072526932
mst_key=2: metric=0.10000000149, loss=14.5062847137
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.10000000149, loss=14.5062856674
mst_key=2: metric=0.10000000149, loss=14.5062847137
CONTEXT: PL/Python function "madlib_keras_automl"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698536437
These 2 methods seems to want `hyperopt` module installed even if I don't call it. It should only error out if I select to use `hyperopt` as an autoML method:
- `madlib_keras_automl`
- `hyperband_schedule`
E.g.,
```
%%sql
DROP TABLE IF EXISTS hb_schedule;
SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);
SELECT * FROM hb_schedule ORDER BY s DESC, i;
```
produces
```
InternalError: (psycopg2.errors.InternalError_) ImportError: No module named hyperopt (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "hyperband_schedule", line 7, in <module>
from deep_learning import madlib_keras_automl
PL/Python function "hyperband_schedule", line 21, in <module>
PL/Python function "hyperband_schedule"
[SQL: SELECT madlib.hyperband_schedule ('hb_schedule',
6,
3,
0);]
```
Same happens with `madlib_keras_automl` when I select `hyperband` as the method I want
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698586366
(4)
`hyperopt` is not learning in the tests I have done so far with `rand`
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=rand',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces no-learning result with labels:
```
INFO: ***Evaluating 3 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 659.287541866 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0988999977708, loss=14.5240154266
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 663.107329845 sec
DETAIL:
Training set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
When I run `hyperband` with same models/hyperparams, it does learn:
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperband',
'R=6, eta=3, skip_last=0',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 655.647205114 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.34889999032, loss=1.83796536922
mst_key=1: metric=0.468120008707, loss=1.481554389
mst_key=2: metric=0.475380003452, loss=1.63904440403
Validation set after iteration 1:
mst_key=3: metric=0.355800002813, loss=1.83082151413
mst_key=1: metric=0.470499992371, loss=1.47972774506
mst_key=2: metric=0.470200002193, loss=1.66988003254
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
Also `hyperopt` with `tpe` is not learning cifar10
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr': [0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr': [0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr': [0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs': [1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=tpe',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute freq
NULL, -- name
NULL); -- descr
```
produces
```
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 648.608314991 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0999400019646, loss=14.5072526932
mst_key=2: metric=0.10000000149, loss=14.5062847137
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.10000000149, loss=14.5062856674
mst_key=2: metric=0.10000000149, loss=14.5062847137
CONTEXT: PL/Python function "madlib_keras_automl"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 edited a comment on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-707417107
Let's be consistent with the output for hyperband and add it to the info table, even if the value will be the same for all mst keys. The table is small so that is not a problem and it may change in the future if we learn something new about how `metrics_iters` should be presented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] fmcquillan99 commented on pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
fmcquillan99 commented on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-699088843
(5)
In this test the 2nd batch is worse than the 1st batch:
```
madlib=# SELECT madlib.madlib_keras_automl('cifar10_train_packed',
madlib(# 'automl_cifar10_output',
madlib(# 'model_arch_library',
madlib(# 'automl_cifar10_mst_table',
madlib(# ARRAY[1,2],
madlib(# $${'loss': ['categorical_crossentropy'],
madlib$# 'optimizer_params_list': [
madlib$# {'optimizer': ['Adam'],'lr': [0.0005, 0.01, 'log']},
madlib$# {'optimizer': ['SGD'],'lr': [0.001, 0.01, 'log'], 'momentum': [0.90, 0.91,'log']}],
madlib$# 'metrics':['accuracy']}$$,
madlib(# $${'batch_size': [64,128], 'epochs': [1]}$$,
madlib(# 'hyperopt',
madlib(# 'num_models=10, num_iters=2, algorithm=tpe',
madlib(# NULL, -- random state
madlib(# NULL, -- object table
madlib(# FALSE, -- use GPUs
madlib(# 'cifar10_val_packed', -- validation table
madlib(# 1, -- metrics compute freq
madlib(# NULL, -- name
madlib(# NULL); -- descr
INFO:
Time for training in iteration 1: 1070.08548617 sec
DETAIL:
Training set after iteration 1:
mst_key=1: metric=0.526019990444, loss=1.5796225071
mst_key=5: metric=0.562780022621, loss=1.37457168102
mst_key=4: metric=0.65314000845, loss=0.969262957573
mst_key=2: metric=0.10000000149, loss=2.30277729034
mst_key=3: metric=0.464760005474, loss=1.50251471996
Validation set after iteration 1:
mst_key=1: metric=0.523299992085, loss=1.60135388374
mst_key=5: metric=0.559000015259, loss=1.40086007118
mst_key=4: metric=0.640600025654, loss=1.00141870975
mst_key=2: metric=0.10000000149, loss=2.30277705193
mst_key=3: metric=0.463400006294, loss=1.50355136395
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 1064.05229688 sec
DETAIL:
Training set after iteration 2:
mst_key=1: metric=0.703299999237, loss=0.840497553349
mst_key=5: metric=0.664420008659, loss=0.98249232769
mst_key=4: metric=0.754859983921, loss=0.701396882534
mst_key=2: metric=0.10000000149, loss=2.30290770531
mst_key=3: metric=0.535520017147, loss=1.26898431778
Validation set after iteration 2:
mst_key=1: metric=0.68900001049, loss=0.903733193874
mst_key=5: metric=0.653800010681, loss=1.02747094631
mst_key=4: metric=0.735899984837, loss=0.765290260315
mst_key=2: metric=0.10000000149, loss=2.30290770531
mst_key=3: metric=0.530799984932, loss=1.28327465057
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Best training loss so far:
DETAIL:
mst_key=4: metric=0.754859983921, loss=0.701396882534
Best validation loss so far:
mst_key=4: metric=0.735899984837, loss=0.765290260315
INFO: ***Evaluating 5 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 1058.9834702 sec
DETAIL:
Training set after iteration 1:
mst_key=6: metric=0.553560018539, loss=1.41088056564
mst_key=10: metric=0.353260010481, loss=1.84440767765
mst_key=9: metric=0.519379973412, loss=1.35283255577
mst_key=7: metric=0.517820000648, loss=1.31637942791
mst_key=8: metric=0.51289999485, loss=1.32810485363
Validation set after iteration 1:
mst_key=6: metric=0.547699987888, loss=1.44565415382
mst_key=10: metric=0.358599990606, loss=1.83988237381
mst_key=9: metric=0.518999993801, loss=1.36131703854
mst_key=7: metric=0.515799999237, loss=1.32058382034
mst_key=8: metric=0.507099986076, loss=1.33919155598
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 1072.08968186 sec
DETAIL:
Training set after iteration 2:
mst_key=6: metric=0.435039997101, loss=1.98741734028
mst_key=10: metric=0.437579989433, loss=1.54891061783
mst_key=9: metric=0.606719970703, loss=1.1007784605
mst_key=7: metric=0.550220012665, loss=1.23340415955
mst_key=8: metric=0.590960025787, loss=1.13847112656
Validation set after iteration 2:
mst_key=6: metric=0.424100011587, loss=2.0407743454
mst_key=10: metric=0.438899993896, loss=1.54700648785
mst_key=9: metric=0.595899999142, loss=1.13602638245
mst_key=7: metric=0.544099986553, loss=1.24483549595
mst_key=8: metric=0.583800017834, loss=1.15895962715
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Best training loss so far:
DETAIL:
mst_key=4: metric=0.754859983921, loss=0.701396882534
Best validation loss so far:
mst_key=4: metric=0.735899984837, loss=0.765290260315
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [madlib] Advitya17 commented on a change in pull request #518: DL: [AutoML] Add support for Hyperopt on top of MOP for MPP + AutoML best-so-far
Posted by GitBox <gi...@apache.org>.
Advitya17 commented on a change in pull request #518:
URL: https://github.com/apache/madlib/pull/518#discussion_r493918741
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -291,9 +338,100 @@ class KerasAutoML():
(self.metrics_compute_frequency >= 1 and \
self.metrics_compute_frequency <= num_iterations)
+ def print_best_so_far(self):
+ """
+ Prints mst keys with best train/val losses at a given point.
+ """
+ best_so_far = '\n'
+ best_so_far += self.print_best_helper('training')
+ if self.validation_table:
+ best_so_far += self.print_best_helper('validation')
+ plpy.info(best_so_far)
+
+ def print_best_helper(self, keyword):
+ """
+ Helper function to Prints mst keys with best train/val losses at a given point.
+ :param keyword: column prefix ('training' or 'validation')
+ :return:
+ """
+ metrics_word, loss_word = keyword + '_metrics_final', keyword + '_loss_final'
+
+ res_str = 'Best {keyword} loss so far:\n'.format(keyword=keyword)
+ best_value = plpy.execute("SELECT {ModelSelectionSchema.MST_KEY}, {metrics_word}, " \
+ "{loss_word} FROM {self.model_info_table} ORDER BY " \
+ "{loss_word} LIMIT 1".format(self=self, ModelSelectionSchema=ModelSelectionSchema,
+ metrics_word=metrics_word, loss_word=loss_word))[0]
+ mst_key_value, metric_value, loss_value = best_value[ModelSelectionSchema.MST_KEY], \
+ best_value[metrics_word], best_value[loss_word]
+ res_str += ModelSelectionSchema.MST_KEY + '=' + str(mst_key_value) + ': metric=' + str(metric_value) + \
+ ', loss=' + str(loss_value) + '\n'
+ return res_str
+
+ def get_current_timestamp(self):
+ """for start and end times for the chosen AutoML algorithm. Showcased in the output summary table"""
+ return datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+ def remove_temp_tables(self, model_training):
+ """
+ Remove all intermediate tables created for AutoML runs/updates.
+ :param model_training: Fit Multiple function call object.
+ """
+ drop_tables([model_training.original_model_output_table, model_training.model_info_table,
+ model_training.model_summary_table, AutoMLSchema.TEMP_MST_TABLE,
+ AutoMLSchema.TEMP_MST_SUMMARY_TABLE])
+
+# @MinWarning("warning")
+class AutoMLHyperband(KerasAutoML):
+ """
+ This class implements Hyperband, an infinite-arm bandit based algorithm that speeds up random search
+ through adaptive resource allocation, successive halving (SHA) and early stopping.
+
+ This class showcases and novel hyperband implementation by executing the hyperband rounds 'diagonally'
Review comment:
I think I just need to add a comma like this: `through adaptive resource allocation, successive halving (SHA), and early stopping`
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -404,24 +544,30 @@ class KerasAutoML():
model_id=ModelSelectionSchema.MODEL_ID,
compile_params=ModelSelectionSchema.COMPILE_PARAMS,
fit_params=ModelSelectionSchema.FIT_PARAMS)
- with MinWarning('warning'):
- plpy.execute(create_query)
+ # with MinWarning('warning'):
Review comment:
See above
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras_automl.py_in
##########
@@ -500,63 +648,432 @@ class KerasAutoML():
"b (key integer, s_val integer, i_val integer) WHERE t.mst_key=b.key".format(self=self, l=l)
plpy.execute(query)
- def update_model_selection_table(self):
+# @MinWarning("warning")
+class AutoMLHyperopt(KerasAutoML):
+ """
+ This class implements Hyperopt, another automl method that explores awkward search spaces using
+ Random Search, Tree-structured Parzen Estimator (TPE), or Adaptive TPE.
+
+ This function executes hyperopt on top of our multiple model training infrastructure powered with
+ Model hOpper Parallelism (MOP), a hybrid of data and task parallelism.
+
+ This automl method inherits qualities from the automl class.
+ """
+ def __init__(self, schema_madlib, source_table, model_output_table, model_arch_table, model_selection_table,
+ model_id_list, compile_params_grid, fit_params_grid, automl_method='hyperopt',
+ automl_params='num_models=20, num_iters=5, algorithm=tpe', random_state=None, object_table=None,
+ use_gpus=False, validation_table=None, metrics_compute_frequency=None,
+ name=None, description=None, **kwargs):
+ KerasAutoML.__init__(self, schema_madlib, source_table, model_output_table, model_arch_table,
+ model_selection_table, model_id_list, compile_params_grid, fit_params_grid,
+ automl_method, automl_params, random_state, object_table, use_gpus,
+ validation_table, metrics_compute_frequency, name, description, **kwargs)
+ self.compile_params_grid = self.compile_params_grid.replace('\n', '').replace(' ', '')
+ self.fit_params_grid = self.fit_params_grid.replace('\n', '').replace(' ', '')
+ try:
+ self.compile_params_grid = literal_eval(self.compile_params_grid)
+
+ except:
+ plpy.error("Invalid syntax in 'compile_params_dict'")
+ try:
+ self.fit_params_grid = literal_eval(self.fit_params_grid)
+ except:
+ plpy.error("Invalid syntax in 'fit_params_dict'")
+ self.validate_and_define_inputs()
+
+ self.num_workers = get_seg_number() * get_segments_per_host()
+
+ self.create_model_output_table()
+ self.create_model_output_info_table()
+ self.find_hyperopt_config()
+
+ def validate_and_define_inputs(self):
+ automl_params_dict = extract_keyvalue_params(self.automl_params,
+ default_values={'num_models': 20,
+ 'num_iters': 5,
+ 'algorithm': 'tpe'},
+ lower_case_names=True)
+ # casting relevant values to int
+ for i in automl_params_dict:
+ try:
+ automl_params_dict[i] = int(automl_params_dict[i])
+ except ValueError:
+ pass
+ _assert(len(automl_params_dict) >= 1 and len(automl_params_dict) <= 3,
+ "DL: Only num_models, num_iters, and algorithm may be specified")
+ for i in automl_params_dict:
+ if i == AutoMLSchema.NUM_MODELS:
+ self.num_models = automl_params_dict[AutoMLSchema.NUM_MODELS]
+ elif i == AutoMLSchema.NUM_ITERS:
+ self.num_iters = automl_params_dict[AutoMLSchema.NUM_ITERS]
+ elif i == AutoMLSchema.ALGORITHM:
+ if automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'rand':
+ self.algorithm = rand
+ elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'tpe':
+ self.algorithm = tpe
+ # elif automl_params_dict[AutoMLSchema.ALGORITHM].lower() == 'atpe':
+ # self.algorithm = atpe
+ # uncomment the above lines after atpe works
+ else:
+ plpy.error("DL: algorithm in 'automl_params' must be 'rand', 'tpe'") # , or 'atpe'
+ else:
+ plpy.error("DL: {0} is an invalid automl param".format(i))
+ _assert(self.num_models > 0 and self.num_iters > 0, "DL: num_models and num_iters in 'automl_params' "
+ "must be positive")
+ _assert(self._is_valid_metrics_compute_frequency(self.num_iters), "DL: 'metrics_compute_frequency' "
+ "out of iteration range")
+
+ def find_hyperopt_config(self):
"""
- Drops and re-create the mst table to only include the best performing model configuration.
+ Executes hyperopt on top of MOP.
"""
- drop_tables([self.model_selection_table])
- # only retaining best performing config
- plpy.execute("CREATE TABLE {self.model_selection_table} AS SELECT mst_key, model_id, compile_params, " \
- "fit_params FROM {self.model_info_table} " \
- "ORDER BY {AutoMLSchema.LOSS_METRIC} LIMIT 1".format(self=self, AutoMLSchema=AutoMLSchema))
+ make_mst_summary = True
+ trials = Trials()
+ domain = Domain(None, self.get_search_space())
+ rand_state = np.random.RandomState(self.random_state)
+ configs_lst = self.get_configs_list()
- def generate_model_output_summary_table(self, model_training):
+ self.start_training_time = self.get_current_timestamp()
+ for low, high in configs_lst:
+ i, n = low, high - low + 1
+
+ # Using HyperOpt TPE/ATPE to generate parameters
+ hyperopt_params = []
+ sampled_params = []
+ for j in range(i, i + n):
+ new_param = self.algorithm.suggest([j], domain, trials, rand_state.randint(0, 2 ** 31 - 1))
+ new_param[0]['status'] = STATUS_RUNNING
+
+ trials.insert_trial_docs(new_param)
+ trials.refresh()
+ hyperopt_params.append(new_param[0])
+ sampled_params.append(new_param[0]['misc']['vals'])
+
+ model_id_list, compile_params, fit_params = self.extract_param_vals(sampled_params)
+ msts_list = self.generate_msts(model_id_list, compile_params, fit_params)
+ try:
+ self.remove_temp_tables(model_training)
+ except:
+ pass
+ self.populate_temp_mst_table(i, msts_list)
+
+ plpy.info("***Evaluating {n} newly suggested model configurations***".format(n=n))
+ model_training = FitMultipleModel(self.schema_madlib, self.source_table, AutoMLSchema.TEMP_OUTPUT_TABLE,
+ AutoMLSchema.TEMP_MST_TABLE, self.num_iters, self.use_gpus, self.validation_table,
+ self.metrics_compute_frequency, False, self.name, self.description)
+ if make_mst_summary:
+ self.generate_mst_summary_table(self.model_selection_summary_table)
+ make_mst_summary = False
+
+ # HyperOpt TPE update
+ for k, hyperopt_param in enumerate(hyperopt_params, i):
+ loss_val = plpy.execute("SELECT {AutoMLSchema.LOSS_METRIC} FROM {model_training.model_info_table} " \
+ "WHERE {ModelSelectionSchema.MST_KEY}={k}".format(AutoMLSchema=AutoMLSchema,
+ ModelSelectionSchema=ModelSelectionSchema,
+ **locals()))[0][AutoMLSchema.LOSS_METRIC]
+
+ hyperopt_param['status'] = STATUS_OK
Review comment:
Yeah it's apparently used internally by the hyperopt module during `refresh`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org