Proposal for Demultiplexing Sink

[Rion Williams <ri...@gmail.com>]

Hi everyone,

I wanted to reach out to the greater Flink audience today to mention a
proposal for a new sink interface to Flink that's an extension of a
previous use-case I've encountered (and perhaps others have as well) which
is a DemultiplexingSink. The core idea is that this sink would work in
conjunction with a separate router interface to dynamically create and
route to sinks at runtime based on characteristics of the message
themselves, something like the example below:

[image: image.png]

The idea was originally presented in FLINK-23977
<https://issues.apache.org/jira/browse/FLINK-23977> and recently
revisited in FLINK-24493 <https://issues.apache.org/jira/browse/FLINK-24493>
based on feedback from several members of the community (especially Fabian
Paul, David Morávek, and Arvid Heise). While the associated draft of what
the PR <https://github.com/apache/flink/pull/27072> might look like isn't
overly large (~10 files, most of which are tests), I believe a contribution
such as this would require the creation of a related FLIP to ensure
adequate discussion, review, and frankly -- to see if there's even interest
in such a project.

The pull request provides a bit more detail on the actual functionality of
the proposed sink and I'm also in the process of writing up a blog post and
associated projects of various examples (e.g., routing to various
Elasticsearch clusters/indices, Kafka topics, files, etc.) that I'd be
happy to share as well.

Looking forward to any feedback!

Thanks,

Rion Williams (rionmonster)