You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Soheil Pourbafrani <so...@gmail.com> on 2020/01/17 13:49:37 UTC

Flink solution for having shared variable between task managers

Hi,

According to the processing logic, I need to have a HashMap variable that
should be shared between the taskmanagers. The scenario is the HashMap data
will be continuously updated according to the incoming stream of data.

What I observed is declaring the HashMap variable as a class attribute, it
will be shared among a single taskmanagers slots, but in case I have
multiple taskmanager, each will have a separate HashMap instance.

What is the standard way to achieve this? Does Flink provide any utility
for that?

RE: EXT :Flink solution for having shared variable between task managers

Posted by "Martin, Nick J [US] (IS)" <Ni...@ngc.com>.

I think you’re looking for Broadcast State. Here’s a detailed guide.

https://flink.apache.org/2019/06/26/broadcast-state.html

From: Soheil Pourbafrani [mailto:soheil.ir08@gmail.com]
Sent: Friday, January 17, 2020 6:50 AM
To: user <us...@flink.apache.org>
Subject: EXT :Flink solution for having shared variable between task managers

Hi,

According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data.

What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance.

What is the standard way to achieve this? Does Flink provide any utility for that?

Re: Flink solution for having shared variable between task managers

Posted by Soheil Pourbafrani <so...@gmail.com>.

Thanks, I'll check it out.

On Mon, Feb 3, 2020 at 10:05 AM Fabian Hueske <fh...@gmail.com> wrote:

> Hi,
>
> I think you are looking for BroadcastState [1].
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html
>
> Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani <
> soheil.ir08@gmail.com>:
>
>> Hi,
>>
>> According to the processing logic, I need to have a HashMap variable that
>> should be shared between the taskmanagers. The scenario is the HashMap data
>> will be continuously updated according to the incoming stream of data.
>>
>> What I observed is declaring the HashMap variable as a class attribute,
>> it will be shared among a single taskmanagers slots, but in case I have
>> multiple taskmanager, each will have a separate HashMap instance.
>>
>> What is the standard way to achieve this? Does Flink provide any utility
>> for that?
>>
>

Re: Flink solution for having shared variable between task managers

Posted by Fabian Hueske <fh...@gmail.com>.

Hi,

I think you are looking for BroadcastState [1].

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html

Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani <
soheil.ir08@gmail.com>:

> Hi,
>
> According to the processing logic, I need to have a HashMap variable that
> should be shared between the taskmanagers. The scenario is the HashMap data
> will be continuously updated according to the incoming stream of data.
>
> What I observed is declaring the HashMap variable as a class attribute, it
> will be shared among a single taskmanagers slots, but in case I have
> multiple taskmanager, each will have a separate HashMap instance.
>
> What is the standard way to achieve this? Does Flink provide any utility
> for that?
>