You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Nathan Deren <na...@zonarsystems.com> on 2017/07/25 22:06:36 UTC

Trouble with ValueProvider/NestedValueProvider

Hi,

I’m using Apache Beam via Google Cloud Dataflow, and I’m working on making a template for my job. My team is working to set up QA and prod environments within separate Google projects, so I’d like to be able to pass the project ID as a parameter at runtime. I’ve been playing around with the ValueProvider<String> type but this causes Java to complain at compile time “java.lang.RuntimeException: Not called from a runtime context.”

I did some googling around and found a few links, and I’ve been working from the official docs as well.
https://stackoverflow.com/questions/43992120/valueprovider-issue
https://stackoverflow.com/questions/42843495/google-dataflow-create-templates-with-runtime-parameters/42843682
https://cloud.google.com/dataflow/docs/templates/creating-templates#using-nestedvalueprovider

The solution seems to be to use the NestedValueProvider type, but I’m still getting a runtime exception if I call .get() on it. I’ve got the following in my pipeline options (according to the Google Cloud Dataflow docs, I’m still supposed to have (Runtime)ValueProviders in my options file):

    ValueProvider<String> getDataflowProjectId();
    void setDataflowProjectId(ValueProvider<String> value);

And I’m composing it into a NestedValueProvider in the following manner:
    NestedValueProvider<String, String> pid = NestedValueProvider.of(
            pipelineOptions.getDataflowProjectId(),
            (SerializableFunction<String, String>) s -> s
    );

If I try following Google’s example and try to assign the output of NestedValueProvider.of directly to a String, I get the following compilation error:

incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider<T,X> conforms to java.lang.String

Java’s not really my language but I’ve been getting along pretty well so far. This however has me completely stumped. What am I doing wrong?

Thanks,
—Nathan Deren

Re: Trouble with ValueProvider/NestedValueProvider

Posted by Nathan Deren <na...@zonarsystems.com>.
To clarify a bit further, this is all happening before pipeline.run() is called – the pipeline is a streaming pipeline intended to run in perpetuity, and we want to do some basic checking of projects, topic IDs, and such before the pipeline starts, but there are several parameter points that differ between the environments and pipeline runs.

From: Nathan Deren
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>"
Date: Tuesday, July 25, 2017 at 3:06 PM
To: "user@beam.apache.org<ma...@beam.apache.org>"
Subject: Trouble with ValueProvider/NestedValueProvider

Hi,

I’m using Apache Beam via Google Cloud Dataflow, and I’m working on making a template for my job. My team is working to set up QA and prod environments within separate Google projects, so I’d like to be able to pass the project ID as a parameter at runtime. I’ve been playing around with the ValueProvider<String> type but this causes Java to complain at compile time “java.lang.RuntimeException: Not called from a runtime context.”

I did some googling around and found a few links, and I’ve been working from the official docs as well.
https://stackoverflow.com/questions/43992120/valueprovider-issue
https://stackoverflow.com/questions/42843495/google-dataflow-create-templates-with-runtime-parameters/42843682
https://cloud.google.com/dataflow/docs/templates/creating-templates#using-nestedvalueprovider

The solution seems to be to use the NestedValueProvider type, but I’m still getting a runtime exception if I call .get() on it. I’ve got the following in my pipeline options (according to the Google Cloud Dataflow docs, I’m still supposed to have (Runtime)ValueProviders in my options file):

    ValueProvider<String> getDataflowProjectId();
    void setDataflowProjectId(ValueProvider<String> value);

And I’m composing it into a NestedValueProvider in the following manner:
    NestedValueProvider<String, String> pid = NestedValueProvider.of(
            pipelineOptions.getDataflowProjectId(),
            (SerializableFunction<String, String>) s -> s
    );

If I try following Google’s example and try to assign the output of NestedValueProvider.of directly to a String, I get the following compilation error:

incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider<T,X> conforms to java.lang.String

Java’s not really my language but I’ve been getting along pretty well so far. This however has me completely stumped. What am I doing wrong?

Thanks,
—Nathan Deren