Fwd: Updating Broadcast Variables

[Pete Wiseman <pw...@semcore.com.au>]

Hi Guys,

Quick question about broadcast variables.

How would one periodically update the value of a broadcast variable?

Suppose my broadcast variable is used for lookups, and these lookup values
can change over time (i.e. some of the lookup values get removed, whilst
others get added). Is it possible to "rebroadcast" it's value?

Or is there some other way to do this kind of thing?

Many thanks,

Pete

Re: Updating Broadcast Variables

[Fabian Hueske <fh...@gmail.com>]

Hi Pete,

Broadcast variables are a feature of the DataSet API [1], i.e., available
for batch processing.
Broadcast variables are computed based on the complete input (which is
possible because they are only available for bounded data sets and not for
unbounded streams) and shared with all operator that require them.
Therefore, there is not need to update a broadcast variable because there
is no more input to update the variable.

I assume that you are looking for a similar feature for the DataStream API.
In the DataStream API, you would use a function with two input, for example
a CoProcessFunction, and broadcast one of the inputs.
Note that with the current version of Flink (1.4.2) you can only connect a
non-keyed stream with a broadcasted stream. The upcoming Flink 1.5.0 will
remove this limitation by introducing broadcasted state.

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/batch/index.html#broadcast-variables

2018-04-04 5:38 GMT+02:00 Pete Wiseman <pw...@semcore.com.au>:

> Hi Guys,
>
> Quick question about broadcast variables.
>
> How would one periodically update the value of a broadcast variable?
>
> Suppose my broadcast variable is used for lookups, and these lookup values
> can change over time (i.e. some of the lookup values get removed, whilst
> others get added). Is it possible to "rebroadcast" it's value?
>
> Or is there some other way to do this kind of thing?
>
> Many thanks,
>
> Pete
>
>
>
>