You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by himanshu singhal <si...@gmail.com> on 2023/08/05 03:01:23 UTC
KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Hello Beam Team,
I am using apache beam for reading from Kafka using KafkaIO in Dataflow
Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
Config, topics) dynamic. I mean, when I am making these parameters as a
PipelineOptions using RuntimeValueProvider My Dataflow Template is not
getting created and getting errors that KafkaIO does not support
RuntimeValueProvider. So in this case can you please suggest or give any
sample code to make these parameters as PipelineOptions.
Thanks & Regards
*Himanshu Singhal*
M:- +917821076244
E:- singhal.himanshu67@gmail.com
Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Posted by Ferran Fernández Garrido <ff...@gmail.com>.
Hi Himanshu,
I would suggest taking a look at this:
https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates
When you create a regular template, it’s pre-configured pipeline. That’s
useful if you always want to apply the same data processes logic. However,
that doesn’t seem to be the case for you. Flex templates, on the other
hand, are an extension that offers flexibility to parameterize the pipeline
code. For the topic, for instance, you can simply use Strings, and there’s
no need for ValueProviders
Best,
Ferran
El El sáb, 5 ago 2023 a las 7:55, himanshu singhal <
singhal.himanshu67@gmail.com> escribió:
> Hello Beam Team,
>
> I am using apache beam for reading from Kafka using KafkaIO in Dataflow
> Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
> Config, topics) dynamic. I mean, when I am making these parameters as a
> PipelineOptions using RuntimeValueProvider My Dataflow Template is not
> getting created and getting errors that KafkaIO does not support
> RuntimeValueProvider. So in this case can you please suggest or give any
> sample code to make these parameters as PipelineOptions.
>
>
>
> Thanks & Regards
> *Himanshu Singhal*
> M:- +917821076244
> E:- singhal.himanshu67@gmail.com
>
Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Posted by himanshu singhal <si...@gmail.com>.
Okay Bruno
Thanks. i will surely try flex templates.
On Fri, Aug 11, 2023, 19:21 Bruno Volpato <bv...@google.com> wrote:
> Hi Himanshu,
>
> I see that Ferran Fernández Garrido provided a very good suggestion.
> For some reason/glitch, I guess you haven't received it.
>
> Please see it in the mail list archive
> https://lists.apache.org/thread/96vz0pdbc6t159vt82ns90qpft1w9d2w
>
>
> On Thu, Aug 10, 2023 at 11:37 AM himanshu singhal <
> singhal.himanshu67@gmail.com> wrote:
>
>> Hello Team,
>> Please update here.
>>
>>
>>
>> Thanks & Regards
>> *Himanshu Singhal*
>> M:- +917821076244 <+91%2078210%2076244>
>> E:- singhal.himanshu67@gmail.com
>>
>>
>> On Sat, Aug 5, 2023 at 8:31 AM himanshu singhal <
>> singhal.himanshu67@gmail.com> wrote:
>>
>>> Hello Beam Team,
>>>
>>> I am using apache beam for reading from Kafka using KafkaIO in Dataflow
>>> Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
>>> Config, topics) dynamic. I mean, when I am making these parameters as a
>>> PipelineOptions using RuntimeValueProvider My Dataflow Template is not
>>> getting created and getting errors that KafkaIO does not support
>>> RuntimeValueProvider. So in this case can you please suggest or give any
>>> sample code to make these parameters as PipelineOptions.
>>>
>>>
>>>
>>> Thanks & Regards
>>> *Himanshu Singhal*
>>> M:- +917821076244 <+91%2078210%2076244>
>>> E:- singhal.himanshu67@gmail.com
>>>
>>
Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Posted by Bruno Volpato via dev <de...@beam.apache.org>.
Hi Himanshu,
I see that Ferran Fernández Garrido provided a very good suggestion.
For some reason/glitch, I guess you haven't received it.
Please see it in the mail list archive
https://lists.apache.org/thread/96vz0pdbc6t159vt82ns90qpft1w9d2w
On Thu, Aug 10, 2023 at 11:37 AM himanshu singhal <
singhal.himanshu67@gmail.com> wrote:
> Hello Team,
> Please update here.
>
>
>
> Thanks & Regards
> *Himanshu Singhal*
> M:- +917821076244 <+91%2078210%2076244>
> E:- singhal.himanshu67@gmail.com
>
>
> On Sat, Aug 5, 2023 at 8:31 AM himanshu singhal <
> singhal.himanshu67@gmail.com> wrote:
>
>> Hello Beam Team,
>>
>> I am using apache beam for reading from Kafka using KafkaIO in Dataflow
>> Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
>> Config, topics) dynamic. I mean, when I am making these parameters as a
>> PipelineOptions using RuntimeValueProvider My Dataflow Template is not
>> getting created and getting errors that KafkaIO does not support
>> RuntimeValueProvider. So in this case can you please suggest or give any
>> sample code to make these parameters as PipelineOptions.
>>
>>
>>
>> Thanks & Regards
>> *Himanshu Singhal*
>> M:- +917821076244 <+91%2078210%2076244>
>> E:- singhal.himanshu67@gmail.com
>>
>
Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Posted by XQ Hu via dev <de...@beam.apache.org>.
I am not entirely sure what you try to do with PipelineOptions. Assuming
you use Beam Java,
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/options/PipelineOptions.html
documents how to construct PipelineOptions is via command-line argument
parsing.
https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Java/src/main/java/kafka/ReadKafka.java
has some code samples.
On Thu, Aug 10, 2023 at 11:37 AM himanshu singhal <
singhal.himanshu67@gmail.com> wrote:
> Hello Team,
> Please update here.
>
>
>
> Thanks & Regards
> *Himanshu Singhal*
> M:- +917821076244 <+91%2078210%2076244>
> E:- singhal.himanshu67@gmail.com
>
>
> On Sat, Aug 5, 2023 at 8:31 AM himanshu singhal <
> singhal.himanshu67@gmail.com> wrote:
>
>> Hello Beam Team,
>>
>> I am using apache beam for reading from Kafka using KafkaIO in Dataflow
>> Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
>> Config, topics) dynamic. I mean, when I am making these parameters as a
>> PipelineOptions using RuntimeValueProvider My Dataflow Template is not
>> getting created and getting errors that KafkaIO does not support
>> RuntimeValueProvider. So in this case can you please suggest or give any
>> sample code to make these parameters as PipelineOptions.
>>
>>
>>
>> Thanks & Regards
>> *Himanshu Singhal*
>> M:- +917821076244 <+91%2078210%2076244>
>> E:- singhal.himanshu67@gmail.com
>>
>
Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam
Posted by himanshu singhal <si...@gmail.com>.
Hello Team,
Please update here.
Thanks & Regards
*Himanshu Singhal*
M:- +917821076244
E:- singhal.himanshu67@gmail.com
On Sat, Aug 5, 2023 at 8:31 AM himanshu singhal <
singhal.himanshu67@gmail.com> wrote:
> Hello Beam Team,
>
> I am using apache beam for reading from Kafka using KafkaIO in Dataflow
> Runner. Here I am facing an issue to make KafkaIO Parameters (like ->
> Config, topics) dynamic. I mean, when I am making these parameters as a
> PipelineOptions using RuntimeValueProvider My Dataflow Template is not
> getting created and getting errors that KafkaIO does not support
> RuntimeValueProvider. So in this case can you please suggest or give any
> sample code to make these parameters as PipelineOptions.
>
>
>
> Thanks & Regards
> *Himanshu Singhal*
> M:- +917821076244
> E:- singhal.himanshu67@gmail.com
>