Stream enrichment with static data, side inputs for DataStream

[Artur Mrozowski <ar...@gmail.com>]

Hi,
I have a stream of buildings and each building has foreign key reference to
municipality. Municipalities data set is quite static. Both are placed on
Kafka topics. I want to enrich each building with municipality name.

FLIP 17, proposal would be ideal for this use case but it's still just a
proposal
https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
.

My question is how do you work around issues like this?

Best regards
Artur

Re: Stream enrichment with static data, side inputs for DataStream

[Artur Mrozowski <ar...@gmail.com>]

Hi Averell,
thank you for your answer.  Indeed it is what I did. And it worked just
fine in this case. I am sure I'll be facing mor complex scenarios, so it'd
be very interesting to hear more about what you try to achieve by using
broadcast state.

Regards

On Thu, Feb 21, 2019 at 11:36 AM Averell <lv...@gmail.com> wrote:

> Hi Artur,
>
> Is that possible to make that "static" stream a keyedStream basing on that
> foreign key?
> If yes, then just connect the two streams, keyed on that foreign key. In
> the
> CoProcessFunction, for every single record from the static stream, you
> write
> its content into a ValueState; and for every record from the main stream,
> you read the enrichment data from the saved ValueState to enrich that
> mainstream record.
>
> If no, then I am having the same issue :D Looking at Broadcast State, but
> there is still something that doesn't look right for me.
>
> Regards,
> Averell
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: Stream enrichment with static data, side inputs for DataStream

[Averell <lv...@gmail.com>]

Hi Artur,

Is that possible to make that "static" stream a keyedStream basing on that
foreign key?
If yes, then just connect the two streams, keyed on that foreign key. In the
CoProcessFunction, for every single record from the static stream, you write
its content into a ValueState; and for every record from the main stream,
you read the enrichment data from the saved ValueState to enrich that
mainstream record.

If no, then I am having the same issue :D Looking at Broadcast State, but
there is still something that doesn't look right for me.

Regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/