You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2020/09/14 23:39:00 UTC

[jira] [Created] (FLINK-19226) [Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high

Hong Liang Teoh created FLINK-19226:
---------------------------------------

             Summary: [Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high
                 Key: FLINK-19226
                 URL: https://issues.apache.org/jira/browse/FLINK-19226
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kinesis
            Reporter: Hong Liang Teoh
             Fix For: 1.12.0


*Background*

When lazily registering the stream consumer on apps with high parallelism, EFO connector hits default maximum number of attempts when calling describeStream and describeStreamConsumer on the Kinesis Streams API.

The default FullJitterBackoff constants are tuned to prevent this when parallelism of 1024 is used.

*Scope*
 * See [FLIP|https://cwiki.apache.org/confluence/display/FLINK/FLIP-128%3A+Enhanced+Fan+Out+for+AWS+Kinesis+Consumers] for full list of configuration options
 * Suggested changes:
|flink.stream.describe.maxretries|50|
|flink.stream.describe.backoff.base|2000L|
|flink.stream.describe.backoff.max|5000L|
|flink.stream.describestreamconsumer.maxretries|50|
|flink.stream.describestreamconsumer.backoff.base|2000L|
|flink.stream.describestreamconsumer.backoff.max|5000L|



--
This message was sent by Atlassian Jira
(v8.3.4#803005)