[jira] [Commented] (MADLIB-1335) Add new param metrics_compute_frequency to madlib_keras_fit()

["Nandish Jayaram (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MADLIB-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831872#comment-16831872 ] 

Nandish Jayaram commented on MADLIB-1335:
-----------------------------------------

*Expected columns in output summary table*:
 # {{metrics_compute_frequency}}, TEXT: The input param that was passed to fit.
 # {{training_loss_final}}, DOUBLE PRECISION: Final iteration's training loss.
 # {{training_metrics_final}} DOUBLE PRECISION []: Final iteration's training metrics.
 # {{validation_loss_final}}, DOUBLE PRECISION: Final iteration's validation loss.
 # {{validation_metrics_final}} DOUBLE PRECISION []: Final iteration's validation metrics.
 # {{training_loss}} DOUBLE PRECISION []: training loss at various iterations determined using {{metrics_compute_frequency}}param.
 # {{training_metrics}} DOUBLE PRECISION []: training metrics at various iterations determined using {{metrics_compute_frequency}} param. Note that this is a 2-D array, where each inner dimension captures all metrics (even if it's just one) for a given iteration.
 # {{validation_loss}} DOUBLE PRECISION []: validation loss at various iterations determined using {{metrics_compute_frequency}} param.
 # {{validation_metrics}} DOUBLE PRECISION []: validation metrics at various iterations determined using {{metrics_compute_frequency}} param. Note that this is a 2-D array, where each inner dimension captures all metrics (even if it's just one) for a given iteration.
 # {{metrics_iters}} DOUBLE PRECISION []: Array containing the various iteration numbers at which loss/metrics were computed for training/validation data.

*NOTE*
 # For this JIRA, limit the scope of {{metrics}} to only {{accuracy}}. Other metrics support to be handled in https://issues.apache.org/jira/browse/MADLIB-1338.
 # {{metrics_compute_frequency}} is an optional param and the default value is NULL. If NULL, then for loss/accuracy over training/validation, only report the {{*_final}} values, and all the iterations related values would be NULL.

> Add new param metrics_compute_frequency to madlib_keras_fit()
> -------------------------------------------------------------
>
>                 Key: MADLIB-1335
>                 URL: https://issues.apache.org/jira/browse/MADLIB-1335
>             Project: Apache MADlib
>          Issue Type: New Feature
>          Components: Deep Learning
>            Reporter: Ekta Khanna
>            Priority: Major
>             Fix For: v1.16
>
>
> JIRA: https://issues.apache.org/jira/browse/MADLIB-1335
> Context
> Getting per iteration loss and other metrics for training and validation data sets can be expensive.  This parameter is intended to give control to user on how often to do this computation.
> Story
> As a data scientist, I want to specify how often to calculate loss and other metrics for training and validation data (if provided), i.e, every n iterations
> Interface
> {code}
> madlib_keras_fit(
>     source_table            VARCHAR,
>     model                   VARCHAR,
>     dependent_varname       VARCHAR,
>     independent_varname     VARCHAR,
>     model_arch_table        VARCHAR,
>     model_arch_id           INTEGER,
>     compile_params          VARCHAR,
>     fit_params              VARCHAR,
>     num_iterations          INTEGER,
>     use_gpu                 BOOLEAN,
>     validation_table        VARCHAR,
>     metrics_compute_frequency      INTEGER   <-------- NEW OPTIONAL PARAM
>     name                    VARCHAR,
>     description             VARCHAR
> {code} 
> where `metrics_compute_frequency` is an optional parameter that means:
> {code}
> NULL  - calculate loss and metrics only on final model after last iteration (default)
> n - calculate loss metrics every n-th iteration and on final model after last iteration
> {code}
> and `metrics_compute_frequency must be >=1 and <=num_iterations`
> Acceptance
> 1) Set `num_iterations = 12` and leave ` metrics_compute_frequency` as default and get 1 loss and metrics value at end after 12 iterations
> 2) Set `num_iterations = 12` and set `metrics_compute_frequency = 5` and get loss and metrics after 5th and 10th iterations and at end after 12 iterations
> 3) Set `metrics_compute_frequency = 0` and get an error
> 4) Set `metrics_compute_frequency = num_iterations+1` and get an error
> 5) Test with different metrics:  from https://keras.io/metrics/ try 'mae' , 'acc' etc.
> Reference
> [1] https://keras.io/metrics/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)