Use case for The Broadcast State Pattern and efficient database access

[Marke Builder <ma...@gmail.com>]

Hi,

I have a question regarding the "Broadcast State Pattern".
My job consume two streams (kafka, rabbitmq), on one of the streams come a
lot of data and continuously[1]. On the other  very few and rarely[2]. I'm
using the Broadcast State pattern, because the stream[2] are updating data
which are required for stream[1].

Q1: Is the Broadcast State Pattern the right way?

As I mentioned above, the stream[2] provide data and "say" read additional
data from a database.

Q2: What is the best(the most efficient) way to request a database from the
processElement(...) function?

Many Thanks!
Marke

Re: Use case for The Broadcast State Pattern and efficient database access

[Andrey Zagrebin <an...@da-platform.com>]

Hi Marke,

Q1: From your description of the problem, "Broadcast State Pattern" seems
to be the suitable choice.
If you want to keep the same state on all parallel instances which process
stream[1] and update/store that state the same way on each instance by
using each element of stream[2].

Q2: Apart of simple synchronous queries to database upon getting each
element of stream[2], you might benefit from using async IO (1). E.g. you
could put it before broadcasting stream[2] and broadcast database response.

Best,
Andrey

(1)
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/asyncio.html

On Thu, Jan 24, 2019 at 1:10 PM Marke Builder <ma...@gmail.com>
wrote:

> Hi,
>
> I have a question regarding the "Broadcast State Pattern".
> My job consume two streams (kafka, rabbitmq), on one of the streams come a
> lot of data and continuously[1]. On the other  very few and rarely[2]. I'm
> using the Broadcast State pattern, because the stream[2] are updating data
> which are required for stream[1].
>
> Q1: Is the Broadcast State Pattern the right way?
>
> As I mentioned above, the stream[2] provide data and "say" read additional
> data from a database.
>
> Q2: What is the best(the most efficient) way to request a database from
> the processElement(...) function?
>
> Many Thanks!
> Marke
>