[jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957813#comment-15957813 ] 

ASF GitHub Bot commented on BEAM-802:
-------------------------------------

GitHub user mariapython opened a pull request:

    https://github.com/apache/beam/pull/2443

    [BEAM-802] Add ValueProvider class for FileBasedSource I/O Transforms

    - [x] Add ValueProvider class.
    - [x] Derive StaticValueProvider and RuntimeValueProvider from ValueProvider.
    - [x] Derive ValueProviderArgumentParser from argparse.ArgumentParser as API for the template user.
    - [x] Modify FileBasedSource I/O transforms to accept objects of type ValueProvider.
    - [x] Modify display_data.
    - [x] Handle serialization / deserialization.
    
    Note: #2441 addresses the failure from the previous version of this PR, and the rest of the issue is tracked in [BEAM-1889].


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mariapython/incubator-beam ppp_fix_equal

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2443
    
----
commit 57838f5be4d1854eec8a51e3aa8e583a82be38d0
Author: Maria Garcia Herrero <ma...@google.com>
Date:   2017-04-05T21:42:13Z

    Revert "Revert "Add ValueProvider class for FileBasedSource I/O Transforms""
    
    This reverts commit 3eef246f761f92c626541e9008f8624b43bdcc09.

----


> Support Dynamic PipelineOptions for python
> ------------------------------------------
>
>                 Key: BEAM-802
>                 URL: https://issues.apache.org/jira/browse/BEAM-802
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: María GH
>            Assignee: María GH
>            Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline options)
> Background: Currently, the Runner creates the JSON pipeline description which can be sent to the worker as is, since everything is already defined there (with links to gs:// for input and binaries). With the parametrized approach, those descriptions are empty and filled by the user or defaulted, so the pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic key/value pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)