You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Benoît Paris (Jira)" <ji...@apache.org> on 2024/10/04 12:46:00 UTC
[jira] [Commented] (FLINK-22748) Allow dynamic target topic selection in SQL Kafka sinks
[ https://issues.apache.org/jira/browse/FLINK-22748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886904#comment-17886904 ]
Benoît Paris commented on FLINK-22748:
--------------------------------------
Well, I'd like to pause and congratulate all the contributors!
First [~twalthr] , [~jark] for the issue management and reviews, [~nicholasjiang] for the first attempt, then [~klam-shop] for picking up the feature again; and finally [~arvid] for the numerous reviews helping this getting merged.
Here we are 3 years later; and I can finally enjoy this quite niche feature. For free thanks to you guys.
I'm quite curious as to its use other than the one I have, now. I'll share it:
This is for distributing reactive web apps. Backend publishes messages from a user to a topic, business rules and event mixing happen in Flink, back it goes to the backend for serving the emergent results to potentially other users. Before this patch I had to make backends read all the messages of a single topic and filter, now they'll register which topic they're going to listen to, and which user they are serving; and Flink will make the routing happen, making a part of my system scalable.
So, what's yours at Celeborn, [~nicholasjiang] ? And yours at Shopify [~klam-shop] ?
> Allow dynamic target topic selection in SQL Kafka sinks
> -------------------------------------------------------
>
> Key: FLINK-22748
> URL: https://issues.apache.org/jira/browse/FLINK-22748
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / Ecosystem
> Reporter: Timo Walther
> Assignee: Kevin Lam
> Priority: Major
> Labels: pull-request-available, stale-assigned, starter
> Fix For: kafka-3.3.0
>
>
> We should allow to write to different Kafka topics based on some column value in SQL.
> The existing implementation can be easily adapted for that. The "target topic" would be an additional persisted metadata column in SQL terms. All one need to do is to adapt
> DynamicKafkaSerializationSchema
> KafkaDynamicSink
> We should guard this dynamic behavior via a config option and make the topic option optional in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)