You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Manninger, Matyas" <ma...@veolia.com> on 2020/12/21 17:16:45 UTC
Session window ad sideinput
Dear Beam users,
I am writing a streaming pipeline that has static tables as side inputs.
These tables change from time to time and I want the side inputs to be
updated at some intervals. I am planning on triggering the update by
sending a message through pubsub. When a new message arrives, the side
input should be updated. I would like to do this with session windows but
on the beam documentation page session windows are depicted as lasting from
the first input in the window to the last input in the window and the gap
seems to not belong to any window. So if I would use this as a side
input how would my main stream be matched to windows? If I send a signal
every day and the gap is set to 1 hour, for example, would the window close
after 1 hour and for the next 23 hours all the elements in the main
stream would be matched to no side input?
Thanks for any help or tips on how to solve this or what is the expected
behaviour.
BR,
Matyas Manninger
Re: Session window ad sideinput
Posted by Reza Ardeshir Rokni <ra...@gmail.com>.
If you can make use of Java SDK, you can use GenerateSequence. With the
python SDK, you should be able to use GenerateSequence via xlang.
Do you have a Jira for the bugs with the PeriodicImpulse?
Cheers
Reza
On Mon, 4 Jan 2021 at 20:12, Manninger, Matyas <ma...@veolia.com>
wrote:
> Dear Reza,
>
> Thanks for the suggestion, that is the solution I was going for but
> unfortunately PeriodicImpulse has some bugs. I also posted a question about
> that in this mail list but no success there so far so I am looking for
> alternatives.
>
> On Tue, 22 Dec 2020 at 12:36, Reza Ardeshir Rokni <ra...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Why the need for session windows? Could you make use of a Global Window
>> for the side input, as per the following pattern:
>>
>> https://beam.apache.org/documentation/patterns/side-inputs/
>>
>> Cheers
>> Reza
>>
>>
>>
>> On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas <
>> matyas.manninger@veolia.com> wrote:
>>
>>> Dear Beam users,
>>>
>>> I am writing a streaming pipeline that has static tables as side inputs.
>>> These tables change from time to time and I want the side inputs to be
>>> updated at some intervals. I am planning on triggering the update by
>>> sending a message through pubsub. When a new message arrives, the side
>>> input should be updated. I would like to do this with session windows but
>>> on the beam documentation page session windows are depicted as lasting from
>>> the first input in the window to the last input in the window and the gap
>>> seems to not belong to any window. So if I would use this as a side
>>> input how would my main stream be matched to windows? If I send a signal
>>> every day and the gap is set to 1 hour, for example, would the window close
>>> after 1 hour and for the next 23 hours all the elements in the main
>>> stream would be matched to no side input?
>>>
>>> Thanks for any help or tips on how to solve this or what is the expected
>>> behaviour.
>>>
>>> BR,
>>> Matyas Manninger
>>>
>>
Re: Session window ad sideinput
Posted by "Manninger, Matyas" <ma...@veolia.com>.
Dear Reza,
Thanks for the suggestion, that is the solution I was going for but
unfortunately PeriodicImpulse has some bugs. I also posted a question about
that in this mail list but no success there so far so I am looking for
alternatives.
On Tue, 22 Dec 2020 at 12:36, Reza Ardeshir Rokni <ra...@gmail.com> wrote:
> Hi,
>
> Why the need for session windows? Could you make use of a Global Window
> for the side input, as per the following pattern:
>
> https://beam.apache.org/documentation/patterns/side-inputs/
>
> Cheers
> Reza
>
>
>
> On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas <
> matyas.manninger@veolia.com> wrote:
>
>> Dear Beam users,
>>
>> I am writing a streaming pipeline that has static tables as side inputs.
>> These tables change from time to time and I want the side inputs to be
>> updated at some intervals. I am planning on triggering the update by
>> sending a message through pubsub. When a new message arrives, the side
>> input should be updated. I would like to do this with session windows but
>> on the beam documentation page session windows are depicted as lasting from
>> the first input in the window to the last input in the window and the gap
>> seems to not belong to any window. So if I would use this as a side
>> input how would my main stream be matched to windows? If I send a signal
>> every day and the gap is set to 1 hour, for example, would the window close
>> after 1 hour and for the next 23 hours all the elements in the main
>> stream would be matched to no side input?
>>
>> Thanks for any help or tips on how to solve this or what is the expected
>> behaviour.
>>
>> BR,
>> Matyas Manninger
>>
>
Re: Session window ad sideinput
Posted by Reza Ardeshir Rokni <ra...@gmail.com>.
Hi,
Why the need for session windows? Could you make use of a Global Window for
the side input, as per the following pattern:
https://beam.apache.org/documentation/patterns/side-inputs/
Cheers
Reza
On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas <ma...@veolia.com>
wrote:
> Dear Beam users,
>
> I am writing a streaming pipeline that has static tables as side inputs.
> These tables change from time to time and I want the side inputs to be
> updated at some intervals. I am planning on triggering the update by
> sending a message through pubsub. When a new message arrives, the side
> input should be updated. I would like to do this with session windows but
> on the beam documentation page session windows are depicted as lasting from
> the first input in the window to the last input in the window and the gap
> seems to not belong to any window. So if I would use this as a side
> input how would my main stream be matched to windows? If I send a signal
> every day and the gap is set to 1 hour, for example, would the window close
> after 1 hour and for the next 23 hours all the elements in the main
> stream would be matched to no side input?
>
> Thanks for any help or tips on how to solve this or what is the expected
> behaviour.
>
> BR,
> Matyas Manninger
>