[jira] [Created] (MADLIB-1333) DL: Add new function for preprocessing images for validation dataset

["Nandish Jayaram (JIRA)" <ji...@apache.org>]

Nandish Jayaram created MADLIB-1333:
---------------------------------------

             Summary: DL: Add new function for preprocessing images for validation dataset
                 Key: MADLIB-1333
                 URL: https://issues.apache.org/jira/browse/MADLIB-1333
             Project: Apache MADlib
          Issue Type: Improvement
          Components: Deep Learning
            Reporter: Nandish Jayaram


Function to prepare the validation dataset for deep learning with madlib
 * This function assumes that the pre processor for training data has already been run.
 * mini-batch x and y.
 * 1-hot encode class levels (for 1-hot) - want to make sure don't miss any class levels (in the case that validation data set by itself does not have all class values that are in the training dataset). This value will be read from the output of the summary table for pre processor for training data.
 * normalizing: use the same normalizing constant that was used while creating batched training data, found in its summary table.
 * rename x and y so that the column names for training data and validation data are the same.
 * applies to fit() and evaluate()

Proposed Interface:
 Rename `minibatch_preprocessor_dl` to `training_preprocessor_dl`. Interface is the same as in master currently:
{code:java}
training_preprocessor_dl( source_table,  -- training dataset
                          output_table,
                          dependent_varname,
                          independent_varname,
                          buffer_size,           	-- Optional
                          normalizing_const,		-- Optional
                          num_classes				-- Optional
                        )
{code}
New function for preparing validation data for evaluation:
{code:java}
validation_preprocessor_dl(
      source_table,  -- validation dataset
      output_table,  
      dependent_varname,
      independent_varname,
      training_preprocessor_table,  -- i.e., from training_preprocessor_dl
      buffer_size           	-- Optional
)
{code}
Acceptance:
 1. Input validation check to ensure `training_preprocessor_table` is not null.
 2. Run `validation_preprocessor_dl` on the exact same data set as `training_preprocessor_dl` and ensure that respective output tables are the same element-by-element. This test may only be verifiable if there was exactly one image in the input table.
3. Make the `buffer_size` in `validation_preprocessor_dl` <1 and ensure fails with nice error message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)