You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexey Trenikhun <ye...@msn.com> on 2021/12/04 00:33:00 UTC
Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules, KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A" (same key) broadcasts "rule 2" with same name X, is there guarantee that eventually broadcast state will contain "rule 2" or since there is no ordering, B could receive "rule 2", then "rule 1" and broadcast state will end up with {X="rule 1"} forever ?
Thanks,
Alexey
Re: Order of events in Broadcast State
Posted by Alexey Trenikhun <ye...@msn.com>.
Thank you David
________________________________
From: David Anderson <da...@apache.org>
Sent: Monday, December 6, 2021 1:36:20 AM
To: Alexey Trenikhun <ye...@msn.com>
Cc: Flink User Mail List <us...@flink.apache.org>
Subject: Re: Order of events in Broadcast State
Event ordering in Flink is only maintained between pairs of events that take exactly the same path through the execution graph. So if you have multiple instances of A (let's call them A1 and A2), each broadcasting a partition of the total rule space, then one instance of B (B1) might receive rule1 from A1 before rule2 from A2, while B2 might receive rule2 before rule1.
If it fits your needs, one simple way to avoid having problems with this is to broadcast from a task with a parallelism of 1. Then every downstream instance will receive the broadcast stream in the same order.
David
On Sat, Dec 4, 2021 at 2:45 AM Alexey Trenikhun <ye...@msn.com>> wrote:
[1] - https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
The Broadcast State Pattern | Apache Flink<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
The Broadcast State Pattern # In this section you will learn about how to use broadcast state in practise. Please refer to Stateful Stream Processing to learn about the concepts behind stateful stream processing. Provided APIs # To show the provided APIs, we will start with an example before presenting their full functionality. As our running example, we will use the case where we have a ...
nightlies.apache.org<http://nightlies.apache.org>
________________________________
From: Alexey Trenikhun <ye...@msn.com>>
Sent: Friday, December 3, 2021 4:33 PM
To: Flink User Mail List <us...@flink.apache.org>>
Subject: Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules, KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A" (same key) broadcasts "rule 2" with same name X, is there guarantee that eventually broadcast state will contain "rule 2" or since there is no ordering, B could receive "rule 2", then "rule 1" and broadcast state will end up with {X="rule 1"} forever ?
Thanks,
Alexey
Re: Order of events in Broadcast State
Posted by David Anderson <da...@apache.org>.
Event ordering in Flink is only maintained between pairs of events that
take exactly the same path through the execution graph. So if you
have multiple instances of A (let's call them A1 and A2), each broadcasting
a partition of the total rule space, then one instance of B (B1) might
receive rule1 from A1 before rule2 from A2, while B2 might receive rule2
before rule1.
If it fits your needs, one simple way to avoid having problems with this is
to broadcast from a task with a parallelism of 1. Then every downstream
instance will receive the broadcast stream in the same order.
David
On Sat, Dec 4, 2021 at 2:45 AM Alexey Trenikhun <ye...@msn.com> wrote:
> [1] -
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
> The Broadcast State Pattern | Apache Flink
> <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
> The Broadcast State Pattern # In this section you will learn about how to
> use broadcast state in practise. Please refer to Stateful Stream Processing
> to learn about the concepts behind stateful stream processing. Provided
> APIs # To show the provided APIs, we will start with an example before
> presenting their full functionality. As our running example, we will use
> the case where we have a ...
> nightlies.apache.org
>
>
> ------------------------------
> *From:* Alexey Trenikhun <ye...@msn.com>
> *Sent:* Friday, December 3, 2021 4:33 PM
> *To:* Flink User Mail List <us...@flink.apache.org>
> *Subject:* Order of events in Broadcast State
>
> Hello,
> Trying to understand what statement "Order of events in Broadcast State
> may differ across tasks" in [1] means.
> Let's say I have keyed function "A" which broadcasting stream of rules,
> KeyedBroadcastProcessFunction "B" receives rules and updates broadcast
> state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X,
> then "A" (same key) broadcasts "rule 2" with same name X, is there
> guarantee that eventually broadcast state will contain "rule 2" or since
> there is no ordering, B could receive "rule 2", then "rule 1" and broadcast
> state will end up with {X="rule 1"} forever ?
>
> Thanks,
> Alexey
>
Re: Order of events in Broadcast State
Posted by Alexey Trenikhun <ye...@msn.com>.
[1] - https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
The Broadcast State Pattern | Apache Flink<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
The Broadcast State Pattern # In this section you will learn about how to use broadcast state in practise. Please refer to Stateful Stream Processing to learn about the concepts behind stateful stream processing. Provided APIs # To show the provided APIs, we will start with an example before presenting their full functionality. As our running example, we will use the case where we have a ...
nightlies.apache.org
________________________________
From: Alexey Trenikhun <ye...@msn.com>
Sent: Friday, December 3, 2021 4:33 PM
To: Flink User Mail List <us...@flink.apache.org>
Subject: Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules, KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A" (same key) broadcasts "rule 2" with same name X, is there guarantee that eventually broadcast state will contain "rule 2" or since there is no ordering, B could receive "rule 2", then "rule 1" and broadcast state will end up with {X="rule 1"} forever ?
Thanks,
Alexey