[jira] [Closed] (MADLIB-1392) DL: Preprocessor support for asymmetric segment distribution

["Orhan Kislal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MADLIB-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Orhan Kislal closed MADLIB-1392.
--------------------------------
    Resolution: Fixed

> DL: Preprocessor support for asymmetric segment distribution
> ------------------------------------------------------------
>
>                 Key: MADLIB-1392
>                 URL: https://issues.apache.org/jira/browse/MADLIB-1392
>             Project: Apache MADlib
>          Issue Type: New Feature
>          Components: Deep Learning
>            Reporter: Ekta Khanna
>            Priority: Major
>             Fix For: v1.17
>
>
> Add asymmetric segment redistribution support to the deep learning preprocessor. Applies to {{training_preprocessor_dl()}} and {{validation_preprocessor_dl()}}
> {code:java}
> training_preprocessor_dl(source_table,
>                          output_table,
>                          dependent_varname,
>                          independent_varname,
>                          buffer_size,
>                          normalizing_const,
>                          num_classes,
>                          distribution_rules    -- new optional param
>                         )
> {code}
> Following are the possible values for the new optional param({{distribution_rules}})
>  # TEXT, *default*: {{all_segments}}. Specifies how to distribute the {{output_table}}. This is important for how the fit function will use resources on the cluster. The default {{all_segments}} means the {{output_table}} will be distributed to all segments in the database cluster.
>  # If you specify {{gpu_segments}} then the {{output_table}} will be distributed to all segments that are on hosts that have GPUs attached. This will make maximum use of GPU resources.
>  # You can also specify the name of a resources table containing the segments to use for training. This table is typically created and maintained by the database administrator. Must contain a column called {{dbid}} that specifies the segment id from the {{gp_segment_configuration}} table.
> Sample {{segments_to_use}} table:
> {code:java}
>  dbid | notes
>  -----|--------------
>     2 | comment here
>     3 | comment here
>     4 | comment here
>     5 | comment here
> {code}
> Same deal as above ^^^ for validation preprocessor.
> This change adds a new column to the output summary table {{gpu_config}}, contains the following values:
> # if {{distribution_policy}} = {{all_segments}}, then {{all_segments}}
> # if {{distribution_policy}} = {{gpu_segments}}, then array of segments ids all segments that are on hosts that have GPUs attached
> # if {{distribution_policy}} = {{segments_to_use_table}}, then array of segments ids, for the above sample {{segments_to_use}} table -> [2,3,4,5]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)