You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Niel Markwick <ni...@google.com> on 2021/08/16 12:18:26 UTC
Re: Use side inputs in beam JDBCIO
JdbcIO requires that the DataSourceConfiguration is specified at pipeline
creation, not runtime.
There is currently no way of specifying a dynamic source/destination.
Therefore you will have to read these parameters from Vault when the
pipeline is being built.
--
<https://cloud.google.com>
* • **Niel Markwick*
* • *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
* • *Google Belgium
* • *nielm@google.com
* • *+32 2 894 6771
Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie.
RPR: 0878.065.378
If you have received this communication by mistake, please don't forward it
to anyone else (it may contain confidential or privileged information),
please erase all copies of it, including all attachments, and please let
the sender know it went to the wrong person. Thanks
On Mon, 16 Aug 2021 at 13:16, Bokolia, Rahul, Vodafone (External) <
rahul.bokolia@vodafone.com> wrote:
> Hi Folks,
>
>
>
> I came to a situation where I need to use some values from side inputs
> inside a class which extends PTransform. At the end I want to create
> DataSourceConfiguration object which takes values like hostname, user name
> and password(these values are coming from Vault and I want to provide them
> as side input to the method that will create DataSourceConfiguration
> object). And the object will be passed to JdbcIO write method like
> mentioned here
> https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/io/jdbc/JdbcIO.html
> .
>
>
>
> But since the JdbcIO does not provide any method that accepts
> ‘Contextful.fn’, I am not able to pass the side inputs.
>
>
>
> Any pointer on how this can be done will be much appreciated!
>
>
>
> Thanks in Advance.
>
>
>
> Thanks and Regards,
>
> Rahul
>
>
>
> C2 General
>
Re: Use side inputs in beam JDBCIO
Posted by Luke Cwik <lc...@google.com>.
Yes an SDF would be great and being able to take DatasourceConfiguration's
as inputs we wouldn't need side inputs within JdbcIO as well.
On Mon, Aug 16, 2021 at 10:07 AM Robert Bradshaw <ro...@google.com>
wrote:
> If JdbcIO was re-written as an SDF, it would be able to take side inputs.
>
> On Mon, Aug 16, 2021 at 5:19 AM Niel Markwick <ni...@google.com> wrote:
>
>> JdbcIO requires that the DataSourceConfiguration is specified at pipeline
>> creation, not runtime.
>> There is currently no way of specifying a dynamic source/destination.
>>
>> Therefore you will have to read these parameters from Vault when the
>> pipeline is being built.
>>
>> --
>> <https://cloud.google.com>
>> * • **Niel Markwick*
>> * • *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
>> * • *Google Belgium
>> * • *nielm@google.com
>> * • *+32 2 894 6771
>>
>>
>> Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: 0878.065.378
>>
>> If you have received this communication by mistake, please don't forward
>> it to anyone else (it may contain confidential or privileged information),
>> please erase all copies of it, including all attachments, and please let
>> the sender know it went to the wrong person. Thanks
>>
>>
>> On Mon, 16 Aug 2021 at 13:16, Bokolia, Rahul, Vodafone (External) <
>> rahul.bokolia@vodafone.com> wrote:
>>
>>> Hi Folks,
>>>
>>>
>>>
>>> I came to a situation where I need to use some values from side inputs
>>> inside a class which extends PTransform. At the end I want to create
>>> DataSourceConfiguration object which takes values like hostname, user name
>>> and password(these values are coming from Vault and I want to provide them
>>> as side input to the method that will create DataSourceConfiguration
>>> object). And the object will be passed to JdbcIO write method like
>>> mentioned here
>>> https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/io/jdbc/JdbcIO.html
>>> .
>>>
>>>
>>>
>>> But since the JdbcIO does not provide any method that accepts
>>> ‘Contextful.fn’, I am not able to pass the side inputs.
>>>
>>>
>>>
>>> Any pointer on how this can be done will be much appreciated!
>>>
>>>
>>>
>>> Thanks in Advance.
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Rahul
>>>
>>>
>>>
>>> C2 General
>>>
>>
Re: Use side inputs in beam JDBCIO
Posted by Robert Bradshaw <ro...@google.com>.
If JdbcIO was re-written as an SDF, it would be able to take side inputs.
On Mon, Aug 16, 2021 at 5:19 AM Niel Markwick <ni...@google.com> wrote:
> JdbcIO requires that the DataSourceConfiguration is specified at pipeline
> creation, not runtime.
> There is currently no way of specifying a dynamic source/destination.
>
> Therefore you will have to read these parameters from Vault when the
> pipeline is being built.
>
> --
> <https://cloud.google.com>
> * • **Niel Markwick*
> * • *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
> * • *Google Belgium
> * • *nielm@google.com
> * • *+32 2 894 6771
>
>
> Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: 0878.065.378
>
> If you have received this communication by mistake, please don't forward
> it to anyone else (it may contain confidential or privileged information),
> please erase all copies of it, including all attachments, and please let
> the sender know it went to the wrong person. Thanks
>
>
> On Mon, 16 Aug 2021 at 13:16, Bokolia, Rahul, Vodafone (External) <
> rahul.bokolia@vodafone.com> wrote:
>
>> Hi Folks,
>>
>>
>>
>> I came to a situation where I need to use some values from side inputs
>> inside a class which extends PTransform. At the end I want to create
>> DataSourceConfiguration object which takes values like hostname, user name
>> and password(these values are coming from Vault and I want to provide them
>> as side input to the method that will create DataSourceConfiguration
>> object). And the object will be passed to JdbcIO write method like
>> mentioned here
>> https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/io/jdbc/JdbcIO.html
>> .
>>
>>
>>
>> But since the JdbcIO does not provide any method that accepts
>> ‘Contextful.fn’, I am not able to pass the side inputs.
>>
>>
>>
>> Any pointer on how this can be done will be much appreciated!
>>
>>
>>
>> Thanks in Advance.
>>
>>
>>
>> Thanks and Regards,
>>
>> Rahul
>>
>>
>>
>> C2 General
>>
>