You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Danny Cranmer (Jira)" <ji...@apache.org> on 2020/10/30 10:24:00 UTC

[jira] [Created] (FLINK-19899) [Kinesis][EFO] Optimise error handling to use a separate exception delivery mechanism

Danny Cranmer created FLINK-19899:
-------------------------------------

             Summary: [Kinesis][EFO] Optimise error handling to use a separate exception delivery mechanism
                 Key: FLINK-19899
                 URL: https://issues.apache.org/jira/browse/FLINK-19899
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kinesis
            Reporter: Danny Cranmer
             Fix For: 1.12.0


*Background*

There is a queue used to pass events between the network client and consumer application. When an error is thrown in the network thread, the queue is cleared to make space for the error event. This means that records will be thrown away to make space for errors (the records would be subsequently reloaded from the shard).

*Scope*

Add a new mechanism to pass exceptions between threads, meaning data does not need to be discarded. When an error is thrown, the error event will be processed by the consumer once all of the records have been processed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)