You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2018/06/22 07:48:00 UTC
[jira] [Resolved] (FLINK-8944) Use ListShards for shard discovery
in the flink kinesis connector
[ https://issues.apache.org/jira/browse/FLINK-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tzu-Li (Gordon) Tai resolved FLINK-8944.
----------------------------------------
Resolution: Fixed
Fix Version/s: 1.6.0
Merged for 1.6.0: ef9e8378929b2f068c71262ed93702da0b023b04
Thanks for the contribution [~kailashhd]
> Use ListShards for shard discovery in the flink kinesis connector
> -----------------------------------------------------------------
>
> Key: FLINK-8944
> URL: https://issues.apache.org/jira/browse/FLINK-8944
> Project: Flink
> Issue Type: Improvement
> Reporter: Kailash Hassan Dayanand
> Assignee: Kailash Hassan Dayanand
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> Currently the DescribeStream AWS API used to get list of shards is has a restricted rate limits on AWS. (5 requests per sec per account). This is problematic when running multiple flink jobs all on same account since each subtasks calls the Describe Stream. Changing this to ListShards will provide more flexibility on rate limits as ListShards has a 100 requests per second per data stream limits.
> More details on the mailing list. https://goo.gl/mRXjKh
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)