KeyedBroadcastProcessFunction

[Sigalit Eliazov <e....@gmail.com>]

Hello
does beam have support for something similar to
KeyedBroadcastProcessFunction which exists in flink?
I am looking for an option to have broadcast state in beam so it can be
shared between different operators

Thanks
Sigalit

Re: KeyedBroadcastProcessFunction

[Sigalit Eliazov <e....@gmail.com>]

About 2gb and it should be distributed

בתאריך יום א׳, 15 במאי 2022, 19:50, מאת Reuven Lax ‏<re...@google.com>:

> How large is this state? Is it distributed?
>
> On Sun, May 15, 2022 at 8:12 AM Sigalit Eliazov <e....@gmail.com>
> wrote:
>
>> Thanks for your response.
>> The use case is 2 pipelines:
>> A. receives information from kafka and "holds" that info
>> B. a pipeline which is triggered by a scheduler and every x minutes
>> should send the info i received in pipeline A to another kafka topic
>>
>> Currently these pipelines are implemented by BEAM and I use DB in order
>> to save the output of pipeline A, so when B is triggered I select the
>> relevant info from DB.
>> I would prefer to drop the DB and use internal state if possible.
>> I also tried to merge those pipelines and use the spec state (@stateId)
>> but it did not work. the state was not available for the operator which
>> reads from the scheduler .
>>
>> Thanks
>> Sigalit
>>
>> On Sun, May 15, 2022 at 6:03 PM Reuven Lax <re...@google.com> wrote:
>>
>>> Beam supports side inputs, which might help you. Can you describe your
>>> use case?
>>>
>>> On Sun, May 15, 2022 at 7:34 AM Sigalit Eliazov <e....@gmail.com>
>>> wrote:
>>>
>>>> Hello
>>>> does beam have support for something similar to
>>>> KeyedBroadcastProcessFunction which exists in flink?
>>>> I am looking for an option to have broadcast state in beam so it can
>>>> be shared between different operators
>>>>
>>>> Thanks
>>>> Sigalit
>>>>
>>>

Re: KeyedBroadcastProcessFunction

[Reuven Lax <re...@google.com>]

How large is this state? Is it distributed?

On Sun, May 15, 2022 at 8:12 AM Sigalit Eliazov <e....@gmail.com> wrote:

> Thanks for your response.
> The use case is 2 pipelines:
> A. receives information from kafka and "holds" that info
> B. a pipeline which is triggered by a scheduler and every x minutes should
> send the info i received in pipeline A to another kafka topic
>
> Currently these pipelines are implemented by BEAM and I use DB in order to
> save the output of pipeline A, so when B is triggered I select the relevant
> info from DB.
> I would prefer to drop the DB and use internal state if possible.
> I also tried to merge those pipelines and use the spec state (@stateId)
> but it did not work. the state was not available for the operator which
> reads from the scheduler .
>
> Thanks
> Sigalit
>
> On Sun, May 15, 2022 at 6:03 PM Reuven Lax <re...@google.com> wrote:
>
>> Beam supports side inputs, which might help you. Can you describe your
>> use case?
>>
>> On Sun, May 15, 2022 at 7:34 AM Sigalit Eliazov <e....@gmail.com>
>> wrote:
>>
>>> Hello
>>> does beam have support for something similar to
>>> KeyedBroadcastProcessFunction which exists in flink?
>>> I am looking for an option to have broadcast state in beam so it can be
>>> shared between different operators
>>>
>>> Thanks
>>> Sigalit
>>>
>>

Re: KeyedBroadcastProcessFunction

[Sigalit Eliazov <e....@gmail.com>]

Thanks for your response.
The use case is 2 pipelines:
A. receives information from kafka and "holds" that info
B. a pipeline which is triggered by a scheduler and every x minutes should
send the info i received in pipeline A to another kafka topic

Currently these pipelines are implemented by BEAM and I use DB in order to
save the output of pipeline A, so when B is triggered I select the relevant
info from DB.
I would prefer to drop the DB and use internal state if possible.
I also tried to merge those pipelines and use the spec state (@stateId) but
it did not work. the state was not available for the operator which reads
from the scheduler .

Thanks
Sigalit

On Sun, May 15, 2022 at 6:03 PM Reuven Lax <re...@google.com> wrote:

> Beam supports side inputs, which might help you. Can you describe your use
> case?
>
> On Sun, May 15, 2022 at 7:34 AM Sigalit Eliazov <e....@gmail.com>
> wrote:
>
>> Hello
>> does beam have support for something similar to
>> KeyedBroadcastProcessFunction which exists in flink?
>> I am looking for an option to have broadcast state in beam so it can be
>> shared between different operators
>>
>> Thanks
>> Sigalit
>>
>

Re: KeyedBroadcastProcessFunction

[Reuven Lax <re...@google.com>]

Beam supports side inputs, which might help you. Can you describe your use
case?

On Sun, May 15, 2022 at 7:34 AM Sigalit Eliazov <e....@gmail.com> wrote:

> Hello
> does beam have support for something similar to
> KeyedBroadcastProcessFunction which exists in flink?
> I am looking for an option to have broadcast state in beam so it can be
> shared between different operators
>
> Thanks
> Sigalit
>