Re: [Proposal] - PubsubIO dynamic topic in DataFlow runner

[tamir yardeni <ta...@gmail.com>]

Hi!

Is it possible adding the topic name in the pubsub message to the internal
DataFlow implementation?

Thanks,
Tamir

‫בתאריך יום ג׳, 28 בדצמ׳ 2021 ב-22:19 מאת ‪Ahmet Altay‬‏ <‪altay@google.com
‬‏>:‬

> Adding people who might be able to comment on : @Reuven Lax
> <re...@google.com> @Dustin Rhodes <dc...@google.com> @Slava Chernyak
> <ch...@google.com>
>
> Most of these people are out of office, and they probably would not
> respond this week.
>
> On Tue, Dec 28, 2021 at 1:29 AM tamir yardeni <ta...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I’m working on a PR to PubsubIO that will allow writing to different
>> topics dynamically [1].
>>
>> PubsubUnboundedSink is replaced by DataFlow runner with internal
>> implementation so it requires changes in the runner.
>>
>> I was able to run the changes in DataFlow by adding the flag
>> “enable_custom_pubsub_sink“ but I guess there is a good reason for the
>> internal replacement.
>>
>> Can someone help with those changes?
>>
>>
>> It was inspired by "BigQuery dynamic destination" and this will have a
>> huge value for my team.
>>
>>
>> [1] - https://github.com/apache/beam/pull/16366
>>
>>
>> Thanks!
>>
>> Tamir
>>
>