You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by zhen li <li...@hotmail.com> on 2018/06/21 02:38:28 UTC
How to use broadcast variables in data stream
Hi,all:
I want to use some other broadcast resources such as list or map in the flatmap function or customer triggers, but I don’t find some api to satisfy.
Anyone can help?
thanks
Re: How to use broadcast variables in data stream
Posted by Rong Rong <wa...@gmail.com>.
Hi Zhen,
This might be a rather inefficient solution. We have encountered situations
that we need to have some daily config update pushed to our flink streaming
application, where the state is very large (but keyed). We end-up having a
service to push that data into a separated kafka stream (which basically
has one burst of messages each day), they used a stream-stream join to
consolidate the configuration with the actual real data stream.
This would only work if your real-stream and your configuration are
basically sharing the same keyed dimension though.
--
Rong
On Sat, Jun 30, 2018 at 2:37 AM zhen li <li...@hotmail.com> wrote:
> Hi，Fabian：
> I use connected stream to solve this problem，one is config stream that
> load some config data from redis.another is real data stream. Due to the
> config data is vary big, then the config stream is slowly than the real
> data stream. When use some config to deal data in flatmap2, it arise the NullPointerException.
> Is there any config to make one stream finished and then start the second?
> or any other ways to solve this problem？
> .
>
> 在 2018年6月21日，下午6:31，Fabian Hueske <fh...@gmail.com> 写道：
>
> That's correct.
>
> Broadcast state was added with Flink 1.5. You can use
> DataStream.broadcast() in Flink 1.3 but it has a few limitations.
> For example, you cannot connect a keyed and a broadcasted stream.
>
> 2018-06-21 11:58 GMT+02:00 zhen li <li...@hotmail.com>:
>
>> Thanks for your reply.
>> But broadcast state seems not supported in Flink-1.3 .
>> I found this in Flink-1.3:
>> Broadcasting
>> DataStream → DataStream
>>
>> Broadcasts elements to every partition.
>>
>> dataStream.broadcast();
>>
>> But I don’t know how to convert it to list and get it in stream context .
>>
>> 在 2018年6月21日，下午5:20，Fabian Hueske <fh...@gmail.com> 写道：
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html
>>
>>
>>
>
>
Re: How to use broadcast variables in data stream
Posted by zhen li <li...@hotmail.com>.
Hi，Fabian：
I use connected stream to solve this problem，one is config stream that load some config data from redis.another is real data stream. Due to the config data is vary big, then the config stream is slowly than the real data stream. When use some config to deal data in flatmap2, it arise the NullPointerException. Is there any config to make one stream finished and then start the second? or any other ways to solve this problem？
.
在 2018年6月21日，下午6:31，Fabian Hueske <fh...@gmail.com>> 写道：
That's correct.
Broadcast state was added with Flink 1.5. You can use DataStream.broadcast() in Flink 1.3 but it has a few limitations.
For example, you cannot connect a keyed and a broadcasted stream.
2018-06-21 11:58 GMT+02:00 zhen li <li...@hotmail.com>>:
Thanks for your reply.
But broadcast state seems not supported in Flink-1.3 .
I found this in Flink-1.3:
Broadcasting
DataStream → DataStream
Broadcasts elements to every partition.
dataStream.broadcast();
But I don’t know how to convert it to list and get it in stream context .
在 2018年6月21日，下午5:20，Fabian Hueske <fh...@gmail.com>> 写道：
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html
Re: How to use broadcast variables in data stream
Posted by Fabian Hueske <fh...@gmail.com>.
That's correct.
Broadcast state was added with Flink 1.5. You can use
DataStream.broadcast() in Flink 1.3 but it has a few limitations.
For example, you cannot connect a keyed and a broadcasted stream.
2018-06-21 11:58 GMT+02:00 zhen li <li...@hotmail.com>:
> Thanks for your reply.
> But broadcast state seems not supported in Flink-1.3 .
> I found this in Flink-1.3:
> Broadcasting
> DataStream → DataStream
>
> Broadcasts elements to every partition.
>
> dataStream.broadcast();
>
> But I don’t know how to convert it to list and get it in stream context .
>
> 在 2018年6月21日，下午5:20，Fabian Hueske <fh...@gmail.com> 写道：
>
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.5/dev/stream/state/broadcast_state.html
>
>
>
Re: How to use broadcast variables in data stream
Posted by zhen li <li...@hotmail.com>.
Thanks for your reply.
But broadcast state seems not supported in Flink-1.3 .
I found this in Flink-1.3:
Broadcasting
DataStream → DataStream
Broadcasts elements to every partition.
dataStream.broadcast();
But I don’t know how to convert it to list and get it in stream context .
在 2018年6月21日，下午5:20，Fabian Hueske <fh...@gmail.com>> 写道：
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html
Re: How to use broadcast variables in data stream
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
if the list is static and not too large, you can pass it as a parameter to
the function.
Function objects are serialized (using Java's default serialization) and
shipped to the workers for execution.
If the data is dynamic, you might want to have a look at Broadcast state
[1].
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html
2018-06-21 4:38 GMT+02:00 zhen li <li...@hotmail.com>:
> Hi,all:
> I want to use some other broadcast resources such as list or map in
> the flatmap function or customer triggers, but I don’t find some api to
> satisfy.
> Anyone can help?
> thanks
>
>
>
>