You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by tamir yardeni <ta...@gmail.com> on 2021/12/28 09:28:44 UTC
[Proposal] - PubsubIO dynamic topic in DataFlow runner
Hi everyone,
I’m working on a PR to PubsubIO that will allow writing to different topics
dynamically [1].
PubsubUnboundedSink is replaced by DataFlow runner with internal
implementation so it requires changes in the runner.
I was able to run the changes in DataFlow by adding the flag
“enable_custom_pubsub_sink“ but I guess there is a good reason for the
internal replacement.
Can someone help with those changes?
It was inspired by "BigQuery dynamic destination" and this will have a huge
value for my team.
[1] - https://github.com/apache/beam/pull/16366
Thanks!
Tamir
Re: [Proposal] - PubsubIO dynamic topic in DataFlow runner
Posted by tamir yardeni <ta...@gmail.com>.
Hi!
Is it possible adding the topic name in the pubsub message to the internal
DataFlow implementation?
Thanks,
Tamir
בתאריך יום ג׳, 28 בדצמ׳ 2021 ב-22:19 מאת Ahmet Altay <altay@google.com
>:
> Adding people who might be able to comment on : @Reuven Lax
> <re...@google.com> @Dustin Rhodes <dc...@google.com> @Slava Chernyak
> <ch...@google.com>
>
> Most of these people are out of office, and they probably would not
> respond this week.
>
> On Tue, Dec 28, 2021 at 1:29 AM tamir yardeni <ta...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I’m working on a PR to PubsubIO that will allow writing to different
>> topics dynamically [1].
>>
>> PubsubUnboundedSink is replaced by DataFlow runner with internal
>> implementation so it requires changes in the runner.
>>
>> I was able to run the changes in DataFlow by adding the flag
>> “enable_custom_pubsub_sink“ but I guess there is a good reason for the
>> internal replacement.
>>
>> Can someone help with those changes?
>>
>>
>> It was inspired by "BigQuery dynamic destination" and this will have a
>> huge value for my team.
>>
>>
>> [1] - https://github.com/apache/beam/pull/16366
>>
>>
>> Thanks!
>>
>> Tamir
>>
>
Re: [Proposal] - PubsubIO dynamic topic in DataFlow runner
Posted by Ahmet Altay <al...@google.com>.
Adding people who might be able to comment on : @Reuven Lax
<re...@google.com> @Dustin Rhodes <dc...@google.com> @Slava Chernyak
<ch...@google.com>
Most of these people are out of office, and they probably would not respond
this week.
On Tue, Dec 28, 2021 at 1:29 AM tamir yardeni <ta...@gmail.com>
wrote:
> Hi everyone,
>
> I’m working on a PR to PubsubIO that will allow writing to different
> topics dynamically [1].
>
> PubsubUnboundedSink is replaced by DataFlow runner with internal
> implementation so it requires changes in the runner.
>
> I was able to run the changes in DataFlow by adding the flag
> “enable_custom_pubsub_sink“ but I guess there is a good reason for the
> internal replacement.
>
> Can someone help with those changes?
>
>
> It was inspired by "BigQuery dynamic destination" and this will have a
> huge value for my team.
>
>
> [1] - https://github.com/apache/beam/pull/16366
>
>
> Thanks!
>
> Tamir
>