[jira] [Commented] (FLINK-33479) Add Rerouting support to KafkaSink

["Kevin Tseng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-33479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783741#comment-17783741 ] 

Kevin Tseng commented on FLINK-33479:
-------------------------------------

can someone assign this to me. we have idea on how this implementation can work and can take it forward if community accept the proposal

> Add Rerouting support to KafkaSink
> ----------------------------------
>
>                 Key: FLINK-33479
>                 URL: https://issues.apache.org/jira/browse/FLINK-33479
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>    Affects Versions: 1.18.0
>            Reporter: Kevin Tseng
>            Priority: Minor
>
> KafkaSink currently operates in a fail / success model. If it fails it will throw FlinkRuntimeException, any record that may cause failure has to be filtered by the operators preceeding KafkaSink. If there's unexpected failure that was caught, this will create continuous failure until a hotfix has been provided and issue addressed.
> Relying on checkpoint to restore the process also create duplicated records produced within the interval, unless EXACTLY_ONCE is utilized.
>  
> Since Flink is meant to be a real-time streaming process, any downtime can create detrimental effect to the downstream consumer of the data.
> I'm proposing to add an error handling injection/intercept to KafkaSink (user can define an error handler or route the error causing record to another downstream operator) to keep the process running. Only throwing RuntimeException when the interceptor was not provided.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)