You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Frank Yellin <fy...@fyellin.com> on 2018/05/04 00:36:34 UTC

How to create a runtime ValueProvider

I'm attempting to create a dataflow template, and within the template have
a variable
    ValueProvider<DateTime> now
such that now is the time the dataflow is started, note the time that the
template was created.

My first attempt was
    ValueProvider<DateTime> now = StaticValueProvider.of(
DateTime.now(DateTimeZone.UTC));

My second attempt was

  public interface MyOptions extends PipelineOptions,
DataflowPipelineOptions {
    @Description("Now")
    @Default.InstanceFactory(GetNow.class)
    ValueProvider<DateTime> getNow();
    void setNow(ValueProvider<DateTime> value);
  }

  static class GetNow implements DefaultValueFactory<DateTime> {
    @Override
    public DateTime create(PipelineOptions options) {
      return DateTime.now(DateTimeZone.UTC);
    }
  }

  ValueProvider<DateTime> now = options.getNow()

My final attempt was:

   ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
        StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));

    ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
x.apply(null));



In every case, it was clear that "now" was being set to template-creation
time rather than actual runtime.

I note that the documentation talks about a RuntimeValueProvider, but there
is no user-visible constructor for this.

Re: How to create a runtime ValueProvider

Posted by Frank Yellin <fy...@fyellin.com>.
Unfortunately, I can't use a side input, because I need to use the result
as an input to a DatastoreV1.Read object.  This is a PTransform rather than
DoFn, so there is no place for a side input.

But you did give me the hint I needed to get something that is close
enough.  It's a bit dirty, but it's what I'll have to use until something
better comes along.

 public interface MyOptions extends PipelineOptions,
DataflowPipelineOptions {
    @Description("ignore")
    @Default.Integer(0)
    ValueProvider<Integer> getIgnored();
    void setIgnored(ValueProvider<Integer> value);
  }

  ValueProvider<DateTime> now =
        NestedValueProvider.of(options.getIgnored(), ignored ->
DateTime.now(DateTimeZone.UTC));

The fact that options.getIgnored() is not static means that the value of
now can't be either.  Each time I call now.get(), the value will be
re-evaluated.  This isn't quite what I wanted, (I was hoping for a single
evaluation), but it will work well enough for my situation.



On Thu, May 3, 2018 at 6:01 PM, Eugene Kirpichov <ki...@google.com>
wrote:

> There is no way to achieve this using ValueProvider. Its value is either
> fixed at construction time (StaticValueProvider), or completely dynamic
> (evaluated every time you call .get()).
> You'll need to implement this using a side input. E.g. take a look at
> implementation of BigQueryIO, how it generates unique job id tokens -
> https://github.com/apache/beam/blob/bf94e36f67a8bc5d24c795e40697ad
> 2504c8594c/sdks/java/io/google-cloud-platform/src/
> main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L756
>
> On Thu, May 3, 2018 at 5:42 PM Frank Yellin <fy...@fyellin.com> wrote:
>
>> [Sorry, I accidentally hit send before I had finished typing . .]
>>
>> Is there any way to achieve what I'm looking for?  Or is this just beyond
>> the scope of ValueProvider and templates?
>>
>>
>>
>> On Thu, May 3, 2018 at 5:36 PM, Frank Yellin <fy...@fyellin.com> wrote:
>>
>>> I'm attempting to create a dataflow template, and within the template
>>> have a variable
>>>     ValueProvider<DateTime> now
>>> such that now is the time the dataflow is started, note the time that
>>> the template was created.
>>>
>>> My first attempt was
>>>     ValueProvider<DateTime> now = StaticValueProvider.of(
>>> DateTime.now(DateTimeZone.UTC));
>>>
>>> My second attempt was
>>>
>>>   public interface MyOptions extends PipelineOptions,
>>> DataflowPipelineOptions {
>>>     @Description("Now")
>>>     @Default.InstanceFactory(GetNow.class)
>>>     ValueProvider<DateTime> getNow();
>>>     void setNow(ValueProvider<DateTime> value);
>>>   }
>>>
>>>   static class GetNow implements DefaultValueFactory<DateTime> {
>>>     @Override
>>>     public DateTime create(PipelineOptions options) {
>>>       return DateTime.now(DateTimeZone.UTC);
>>>     }
>>>   }
>>>
>>>   ValueProvider<DateTime> now = options.getNow()
>>>
>>> My final attempt was:
>>>
>>>    ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
>>>         StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));
>>>
>>>     ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
>>> x.apply(null));
>>>
>>>
>>>
>>> In every case, it was clear that "now" was being set to
>>> template-creation time rather than actual runtime.
>>>
>>> I note that the documentation talks about a RuntimeValueProvider, but
>>> there is no user-visible constructor for this.
>>>
>>>
>>>
>>>
>>

Re: How to create a runtime ValueProvider

Posted by Eugene Kirpichov <ki...@google.com>.
There is no way to achieve this using ValueProvider. Its value is either
fixed at construction time (StaticValueProvider), or completely dynamic
(evaluated every time you call .get()).
You'll need to implement this using a side input. E.g. take a look at
implementation of BigQueryIO, how it generates unique job id tokens -
https://github.com/apache/beam/blob/bf94e36f67a8bc5d24c795e40697ad2504c8594c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L756


On Thu, May 3, 2018 at 5:42 PM Frank Yellin <fy...@fyellin.com> wrote:

> [Sorry, I accidentally hit send before I had finished typing . .]
>
> Is there any way to achieve what I'm looking for?  Or is this just beyond
> the scope of ValueProvider and templates?
>
>
>
> On Thu, May 3, 2018 at 5:36 PM, Frank Yellin <fy...@fyellin.com> wrote:
>
>> I'm attempting to create a dataflow template, and within the template
>> have a variable
>>     ValueProvider<DateTime> now
>> such that now is the time the dataflow is started, note the time that the
>> template was created.
>>
>> My first attempt was
>>     ValueProvider<DateTime> now =
>> StaticValueProvider.of(DateTime.now(DateTimeZone.UTC));
>>
>> My second attempt was
>>
>>   public interface MyOptions extends PipelineOptions,
>> DataflowPipelineOptions {
>>     @Description("Now")
>>     @Default.InstanceFactory(GetNow.class)
>>     ValueProvider<DateTime> getNow();
>>     void setNow(ValueProvider<DateTime> value);
>>   }
>>
>>   static class GetNow implements DefaultValueFactory<DateTime> {
>>     @Override
>>     public DateTime create(PipelineOptions options) {
>>       return DateTime.now(DateTimeZone.UTC);
>>     }
>>   }
>>
>>   ValueProvider<DateTime> now = options.getNow()
>>
>> My final attempt was:
>>
>>    ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
>>         StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));
>>
>>     ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
>> x.apply(null));
>>
>>
>>
>> In every case, it was clear that "now" was being set to template-creation
>> time rather than actual runtime.
>>
>> I note that the documentation talks about a RuntimeValueProvider, but
>> there is no user-visible constructor for this.
>>
>>
>>
>>
>

Re: How to create a runtime ValueProvider

Posted by Frank Yellin <fy...@fyellin.com>.
[Sorry, I accidentally hit send before I had finished typing . .]

Is there any way to achieve what I'm looking for?  Or is this just beyond
the scope of ValueProvider and templates?



On Thu, May 3, 2018 at 5:36 PM, Frank Yellin <fy...@fyellin.com> wrote:

> I'm attempting to create a dataflow template, and within the template have
> a variable
>     ValueProvider<DateTime> now
> such that now is the time the dataflow is started, note the time that the
> template was created.
>
> My first attempt was
>     ValueProvider<DateTime> now = StaticValueProvider.of(DateTim
> e.now(DateTimeZone.UTC));
>
> My second attempt was
>
>   public interface MyOptions extends PipelineOptions,
> DataflowPipelineOptions {
>     @Description("Now")
>     @Default.InstanceFactory(GetNow.class)
>     ValueProvider<DateTime> getNow();
>     void setNow(ValueProvider<DateTime> value);
>   }
>
>   static class GetNow implements DefaultValueFactory<DateTime> {
>     @Override
>     public DateTime create(PipelineOptions options) {
>       return DateTime.now(DateTimeZone.UTC);
>     }
>   }
>
>   ValueProvider<DateTime> now = options.getNow()
>
> My final attempt was:
>
>    ValueProvider<SerializableFunction<Void, DateTime>> nowFn =
>         StaticValueProvider.of(x -> DateTime.now(DateTimeZone.UTC));
>
>     ValueProvider<DateTime> now = NestedValueProvider.of(nowFn, x ->
> x.apply(null));
>
>
>
> In every case, it was clear that "now" was being set to template-creation
> time rather than actual runtime.
>
> I note that the documentation talks about a RuntimeValueProvider, but
> there is no user-visible constructor for this.
>
>
>
>