[jira] [Created] (BEAM-551) Support Dynamic PipelineOptions

["Sam McVeety (JIRA)" <ji...@apache.org>]

Sam McVeety created BEAM-551:
--------------------------------

             Summary: Support Dynamic PipelineOptions
                 Key: BEAM-551
                 URL: https://issues.apache.org/jira/browse/BEAM-551
             Project: Beam
          Issue Type: New Feature
          Components: beam-model
            Reporter: Sam McVeety
            Assignee: Frances Perry
            Priority: Minor


During the graph construction phase, the given SDK generates an initial
execution graph for the program.  At execution time, this graph is
executed, either locally or by a service.  Currently, Beam only supports
parameterization at graph construction time.  Both Flink and Spark supply
functionality that allows a pre-compiled job to be run without SDK
interaction with updated runtime parameters.

In its current incarnation, Dataflow can read values of PipelineOptions at
job submission time, but this requires the presence of an SDK to properly
encode these values into the job.  We would like to build a common layer
into the Beam model so that these dynamic options can be properly provided
to jobs.

Please see
https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
for the high-level model, and
https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
for
the specific API proposal.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)