Multiple operators to the same sink

[burgesschen <tc...@bloomberg.net>]

Hi Guys! I'm designing a topology where multiple operators should forward the
messages to the same sink.


For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and
D, E to forward to Sink2.

My options are

1. Union A, B and C. then add Sink1 to them. Similarly for D and E. However,
the current framework out team has builds each operator individually. There
is nothing outside of the operators 
that has the knowledge of their destination sink. It means we need to build
something on the job level to union the operators.

2. have each operator output to a side output tag. A,B, and C will output to
tag "sink1", And have a singleton sink1 to consume from tag "sink1".
Similarly for sink2. My concern here is that 'it feels hacky', since those
messages are not really side outputs.

is this a legitimate use case for output tag or not? Is there a better way
to achieve this? Thank you!






--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Multiple operators to the same sink

[Timo Walther <tw...@apache.org>]

Hi,

I'm not quite sure if I understand your problem correctly. But your use 
case sounds like a typical application of a union operation.

What do you mean with "knowledge of their destination sink"? The 
operators don't need to be aware of the destination sink. The only thing 
that needs to be coordinated is the result data type of each operation. 
So either you force each operation to have a unified type or you create 
a unified type before declaring the sink.

Or is every operator an independent Flink job? Maybe you can show us a 
skeleton of your pipeline?

Regards,
Timo


Am 08.11.18 um 01:20 schrieb burgesschen:
> Hi Guys! I'm designing a topology where multiple operators should forward the
> messages to the same sink.
>
>
> For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and
> D, E to forward to Sink2.
>
> My options are
>
> 1. Union A, B and C. then add Sink1 to them. Similarly for D and E. However,
> the current framework out team has builds each operator individually. There
> is nothing outside of the operators
> that has the knowledge of their destination sink. It means we need to build
> something on the job level to union the operators.
>
> 2. have each operator output to a side output tag. A,B, and C will output to
> tag "sink1", And have a singleton sink1 to consume from tag "sink1".
> Similarly for sink2. My concern here is that 'it feels hacky', since those
> messages are not really side outputs.
>
> is this a legitimate use case for output tag or not? Is there a better way
> to achieve this? Thank you!
>
>
>
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/