You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Smirnov Sergey Vladimirovich <s....@tinkoff.ru> on 2019/06/06 11:50:42 UTC
Change sink topology
Hi flink,
Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?
Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).
Simplifying: sink_topic = 'logger_group_'+message.groupId
When job was launched we don`t know list of all possible groupId.
These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.
Best, Sergey
RE: Change sink topology
Posted by Smirnov Sergey Vladimirovich <s....@tinkoff.ru>.
Great, thanks!
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Thursday, June 6, 2019 3:07 PM
To: Smirnov Sergey Vladimirovich <s....@tinkoff.ru>
Cc: user@flink.apache.org
Subject: Re: Change sink topology
Hi Sergey,
I would not consider this to be a topology change (the sink operator would still be a Kafka producer).
It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).
Best,
Fabian
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer
Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <s....@tinkoff.ru>>:
Hi flink,
Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly?
Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property).
Simplifying: sink_topic = ‘logger_group_’+message.groupId
When job was launched we don`t know list of all possible groupId.
These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period.
Best, Sergey
Re: Change sink topology
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Sergey,
I would not consider this to be a topology change (the sink operator would
still be a Kafka producer).
It seems that dynamic topic selection is possible with a
KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).
Best,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer
Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <
s.smirnov1@tinkoff.ru>:
> Hi flink,
>
>
>
> Im wonder, is it possible to dynamically (while job running) change sink
> topology* - by adding new sink on the fly?
>
> Say, we have input stream and by analyzing message property we decided to
> put this message into some kafka topic, i.e. choosen_topic =
> function(message.property).
>
> Simplifying: sink_topic = ‘logger_group_’+message.groupId
>
> When job was launched we don`t know list of all possible groupId.
>
> These sink topics (and groupId) are created (and even removed) dynamically
> upon executing some external rules during the entire job period.
>
>
>
>
>
> Best, Sergey
>