[jira] [Commented] (FLINK-24493) Introduce DemultiplexingSink to Support Dynamic Sink Routing

["Rion Williams (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18023929#comment-18023929 ] 

Rion Williams commented on FLINK-24493:
---------------------------------------

[~fpaul] / [~arvid] / [~dmvk] 

I've created a draft pull request for the initial pass for this effort available below:
 * [https://github.com/apache/flink/pull/27072]

There's still quite a bit of work and improvements to make, but I feel pretty good about this initial pass. The ones that immediately come to mind include:
 * Address currently shimmed out implementation for `DemultiplexingSinkWriter.serializeWriterStates` to ensure that checkpointing behavior and recovery scenarios are accounted for.
 ** Extend existing test suite to verify these and potential edge-cases that may arise related to them.
 * Add Test Container-backed integration test suite to verify expected behavior for popular and widely supported sinks (e.g. Kafka, JDBC, Elasticsearch, etc.)  
 * Additional documentation and examples

I'd appreciate any and all feedback as it's been a while but since the recent migration to the Sink API from my initial implementation is fairly fresh in my mind, I figured it was a better time than any to try and contribute.

Thanks again,

Rion

> Introduce DemultiplexingSink to Support Dynamic Sink Routing
> ------------------------------------------------------------
>
>                 Key: FLINK-24493
>                 URL: https://issues.apache.org/jira/browse/FLINK-24493
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Common
>            Reporter: Rion Williams
>            Assignee: Rion Williams
>            Priority: Major
>              Labels: pull-request-available
>
> Recently, FLINK-23977 attempted to introduce an approach to supporting dynamic routing for the Elasticsearch sink, however during some discussion within [the pull request|https://github.com/apache/flink/pull/17061], the idea was introduced to create a more generic approach.
> The idea being that we could introduce a common DemultiplexingSink and related interface for handling routing to any number of existing sinks similar to the implementation mentioned in FLINK-23977 to the common connectors directory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)