You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Smirnov Sergey Vladimirovich <s....@tinkoff.ru> on 2019/06/06 11:50:42 UTC

Change sink topology

Hi flink,

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?
Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).
Simplifying: sink_topic = 'logger_group_'+message.groupId
When job was launched we don`t know list of all possible groupId.
These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.


Best, Sergey

RE: Change sink topology

Posted by Smirnov Sergey Vladimirovich <s....@tinkoff.ru>.
Great, thanks!

From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Thursday, June 6, 2019 3:07 PM
To: Smirnov Sergey Vladimirovich <s....@tinkoff.ru>
Cc: user@flink.apache.org
Subject: Re: Change sink topology

Hi Sergey,

I would not consider this to be a topology change (the sink operator would still be a Kafka producer).
It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

Best,
Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <s....@tinkoff.ru>>:
Hi flink,

Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?
Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).
Simplifying: sink_topic = ‘logger_group_’+message.groupId
When job was launched we don`t know list of all possible groupId.
These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.


Best, Sergey

Re: Change sink topology

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Sergey,

I would not consider this to be a topology change (the sink operator would
still be a Kafka producer).
It seems that dynamic topic selection is possible with a
KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

Best,
Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <
s.smirnov1@tinkoff.ru>:

> Hi flink,
>
>
>
> Im wonder, is it possible to dynamically (while job running) change sink
> topology* - by adding new sink on the fly?
>
> Say, we have input stream and by analyzing message property we decided to
> put this message into some kafka topic, i.e. choosen_topic =
> function(message.property).
>
> Simplifying: sink_topic = ‘logger_group_’+message.groupId
>
> When job was launched we don`t know list of all possible groupId.
>
> These sink topics (and groupId) are created (and even removed) dynamically
> upon executing some external rules during the entire job period.
>
>
>
>
>
> Best, Sergey
>