You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Rion Williams (Jira)" <ji...@apache.org> on 2021/10/14 19:53:00 UTC
[jira] [Comment Edited] (FLINK-24493) Introduce DemultiplexingSink
to Support Dynamic Sink Routing
[ https://issues.apache.org/jira/browse/FLINK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428994#comment-17428994 ]
Rion Williams edited comment on FLINK-24493 at 10/14/21, 7:52 PM:
------------------------------------------------------------------
[~fabian.paul] / [~dmvk] / [~arvid]
I had a bit of time to think about this earlier today and I'm excited about working on it. I did have a handful of questions that one or more of you folks might be able to help with:
* I was curious if there was any documentation / examples / tests that I could use to glean from with regards to the new Sink interface. I'm not terribly familiar with it as much as the previous common uses of SinkFunctions, etc. Could anyone point me in a direction on that front?
* Given the changes made to the Sink interface, would the expectation be that this new DemultiplexingSink inherits from a RichSinkFunction similar to my previous implementation for the use-case for Elasticsearch? Or would it just directly implement the Sink interface and the router would govern how to construct the sink itself [similar to the port of the Elasticsearch 7 sink|https://github.com/apache/flink/pull/17363/files].
* As a follow up to the previous one, I'd imagine this demultiplexer could conceivably route records to multiple different classes of Sink (e.g. send records to Elasticsearch, send records to Kafka, JDBC, etc.). Is it out of the question for a single record to be sent to multiple different sinks (send to all n different routes it knows about)? If so, I'd imagine that the genericizing of the Sink type parameters might be tricky? Maybe not?
I'll keep digging around a bit - but I think after looking over the ES7 implementation (while writing this) might answer most of these questions since it seems to easily show how the older interfaces compare to the new one (e.g. SinkFunction.invoke() vs Sink.emitter).
Thanks folks! I'm hoping to dig to this a bit more next week as I'll be participating in the trainings offered at Flink Forward and looking forward to the sessions at the conference itself.
Rion
(P.S. How would one get a Flink shirt if they exist?)
was (Author: rionmonster):
[~fabian.paul] / [~dmvk] / [~arvid]
I had a bit of time to think about this earlier today and I'm excited about working on it. I did have a handful of questions that one or more of you folks might be able to help with:
* I was curious if there was any documentation / examples / tests that I could use to glean from with regards to the new Sink interface. I'm not terribly familiar with it as much as the previous common uses of SinkFunctions, etc. Could anyone point me in a direction on that front?
* Given the changes made to the Sink interface, would the expectation be that this new DemultiplexingSink inherits from a RichSinkFunction similar to my previous implementation for the use-case for Elasticsearch? Or would it just directly implement the Sink interface and the router would govern how to construct the sink itself [similar to the port of the Elasticsearch 7 sink|https://github.com/apache/flink/pull/17363/files].
I'll keep digging around a bit - but I think after looking over the ES7 implementation (while writing this) might answer most of these questions since it seems to easily show how the older interfaces compare to the new one (e.g. SinkFunction.invoke() vs Sink.emitter).
Thanks folks! I'm hoping to dig to this a bit more next week as I'll be participating in the trainings offered at Flink Forward and looking forward to the sessions at the conference itself.
Rion
(P.S. How would one get a Flink shirt if they exist?)
> Introduce DemultiplexingSink to Support Dynamic Sink Routing
> ------------------------------------------------------------
>
> Key: FLINK-24493
> URL: https://issues.apache.org/jira/browse/FLINK-24493
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common
> Reporter: Rion Williams
> Assignee: Rion Williams
> Priority: Major
>
> Recently, FLINK-23977 attempted to introduce an approach to supporting dynamic routing for the Elasticsearch sink, however during some discussion within [the pull request|https://github.com/apache/flink/pull/17061], the idea was introduced to create a more generic approach.
> The idea being that we could introduce a common DemultiplexingSink and related interface for handling routing to any number of existing sinks similar to the implementation mentioned in FLINK-23977 to the common connectors directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)