Re: Providing default values for ValueProvider

[Kenneth Knowles <kl...@google.com>]

It sounds like it may simply be an oversight that @Default does not work
with ValueProvider. Copying dev@ to see if there's some pitfall associated
with adding this.

Kenn

On Thu, Dec 20, 2018 at 5:17 PM Jeff Klukas <jk...@mozilla.com> wrote:

> I am also now realizing that runtime params show up in --help with type of
> ValueProvider, so I can likely meet my documentation needs via a README for
> the project that explains how runtime parameters behave and that
> ValueProvider type is the flag that lets you know a parameter can be
> overridden at runtime.
>
>
>
> On Thu, Dec 20, 2018 at 4:47 PM Jeff Klukas <jk...@mozilla.com> wrote:
>
>> I did some more testing this afternoon, and it looks like I'm incorrect
>> on some details here.
>>
>> Parameters implemented as ValueProviders can be provided at compile time
>> and they do end up in the template. A value specified at runtime overrides.
>> If not specified at runtime, the compile-time value is in effect.
>>
>> It still looks, though, like @Default has no effect for ValueProviders.
>> If not value is specified at compile time or runtime, the ValueProvider is
>> not available, even if it has a @Default configured.
>>
>> Does anybody have tips for documenting which params are overrideable at
>> runtime? Or experience handling default values for ValueProviders?
>>
>> On Thu, Dec 20, 2018 at 3:49 PM Jeff Klukas <jk...@mozilla.com> wrote:
>>
>>> I am deploying Beam pipelines with the DataflowRunner and would like to
>>> move more of the pipeline options to use the ValueProvider interface so
>>> they can be specified at runtime rather than at template compile time, but
>>> running into various issues.
>>>
>>> First, it's been unclear to the operations engineers deploying pipelines
>>> which options are runtime and which are compile-time. The engineers are
>>> typically using the gcloud command-line interface to deploy rather than the
>>> console, so I don't see much benefit from implementing a metadata json file.
>>>
>>> AFAICT, Dataflow happily accepts values for runtime parameters when
>>> compiling the template, but those values are completely ignored and need to
>>> be defined again at runtime, furthering the confusion. Should I just go
>>> ahead and add "(runtime parameter)" to each of the @Description annotations
>>> to at least document the distinction in --help output?
>>>
>>> Finally, it's not clear whether the @Default annotation supports runtime
>>> parameters. The dataflow docs show an example where @Default is used on a
>>> ValueProvider [0], but this doesn't appear to actually have any effect. If
>>> I don't pass in a value for a runtime parameter when executing a template,
>>> the pipeline throws a "Value only available at runtime" exception on calls
>>> to .get(), rather than returning the default value. Have others encountered
>>> this? Is there a pattern for providing defaults for runtime parameters?
>>>
>>> [0]
>>> https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#using-valueprovider-in-your-pipeline-options
>>>
>>