Flink streaming connect and split streams

[Martin Neumann <mn...@sics.se>]

Hej,

I'm currently playing around with some machine learning algorithms in Flink
streaming.

I have an input stream that I partition by key and then do a map on each of
the keys, feeding a model and producing a prediction output. Periodically
each operator needs to send model updates to all other operators.

What is the best way to implement the structure?

My current idea is to use the CoMap function as operator. The first stream
is the raw data the second stream the model updates which I could just
broadcast from the iterative stream. My problem right now is that I need
the CoMap to basically have 2 Streams as output the model updates and the
prediction results.

I could write a wrapper class containing both output types but that would
require me to separate them afterwards. This feels very clunky, is there a
better way of dealing with this?

cheers Martin

Re: Flink streaming connect and split streams

[Aljoscha Krettek <al...@apache.org>]

Hi,
I'm afraid the only way to do it right now is using the wrapper that can
contain both, as you suggested.

Cheers,
Aljoscha

On Thu, 30 Jun 2016 at 16:50 Martin Neumann <mn...@sics.se> wrote:

> Hej,
>
> I'm currently playing around with some machine learning algorithms in
> Flink streaming.
>
> I have an input stream that I partition by key and then do a map on each
> of the keys, feeding a model and producing a prediction output.
> Periodically each operator needs to send model updates to all other
> operators.
>
> What is the best way to implement the structure?
>
> My current idea is to use the CoMap function as operator. The first stream
> is the raw data the second stream the model updates which I could just
> broadcast from the iterative stream. My problem right now is that I need
> the CoMap to basically have 2 Streams as output the model updates and the
> prediction results.
>
> I could write a wrapper class containing both output types but that would
> require me to separate them afterwards. This feels very clunky, is there a
> better way of dealing with this?
>
> cheers Martin
>