[jira] [Created] (FLINK-10536) Flink Kinesis Consumer leading to job failure due to ProvisionedThroughputExceededException

["Ravi Bhushan Ratnakar (JIRA)" <ji...@apache.org>]

Ravi Bhushan Ratnakar created FLINK-10536:
---------------------------------------------

             Summary: Flink Kinesis Consumer leading to job failure due to ProvisionedThroughputExceededException
                 Key: FLINK-10536
                 URL: https://issues.apache.org/jira/browse/FLINK-10536
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
    Affects Versions: 1.5.2
            Reporter: Ravi Bhushan Ratnakar


*Background:*

Flink Kinesis consumer, while consuming message from a shard, it retries for a configured number of attempts and then after it throws "ProvisionedThroughputExceededException" and which bubbles up to top level and eventually which leads to failure of the running streaming job.

*Improvement:*

In some scenario it may be desirable that to skip some of the records by skipping that "shardIterator" and move to next iterator in "ShardConsumer" and log this as metrics. This can be configurable through some property. This way those application which can tolerate to skip some messages can continue to keep running.

 

If this idea seems useful, i would like to contribute on this. Please let me know the process

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)