You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Hong Liang Teoh (Jira)" <ji...@apache.org> on 2024/11/04 22:03:00 UTC
[jira] [Updated] (FLINK-36637) Fix ListShards for Kinesis streams with large number of shards
[ https://issues.apache.org/jira/browse/FLINK-36637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Liang Teoh updated FLINK-36637:
------------------------------------
Fix Version/s: aws-connector-5.0.0
> Fix ListShards for Kinesis streams with large number of shards
> --------------------------------------------------------------
>
> Key: FLINK-36637
> URL: https://issues.apache.org/jira/browse/FLINK-36637
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
> Affects Versions: aws-connector-4.4.0
> Reporter: Elphas Toringepi
> Assignee: Elphas Toringepi
> Priority: Critical
> Labels: pull-request-available
> Fix For: aws-connector-4.4.0, aws-connector-5.0.0
>
>
> The new Kinesis source fails to consume from streams with a large number of shards because it cannot successfully list the shards. This issue was discovered on a stream with 1024 shards.
> The following exception is thrown:
> {{org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: Kinesis Stream Source' (operator bc764cd8ddf7a0cff126f51c16239658).}}
> {{ at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)}}
> {{ ...}}
> {{Caused by: org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException: Failed to list shards.}}
> {{ at org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.processDiscoveredSplits(KinesisStreamsSourceEnumerator.java:375)}}
> {{ at org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$1(ExecutorNotifier.java:83)}}
> {{ at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)}}
> {{ ... 6 more}}
> {{Caused by: org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException: Unable to list shards}}
> {{ at org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.initialDiscoverSplits(KinesisStreamsSourceEnumerator.java:260)}}
> {{ at org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)}}
> {{ ... 6 more}}
> {{Caused by: org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: NextToken and StreamARN cannot be provided together. (Service: Kinesis, Status Code: 400, Request ID: d0ca3ae9-975e-6f46-8f56-70aa948cfa30, Extended Request ID: +4y3p3zgJ9MOZ7FOL4C1LnhrVm69QLZXbwvCyZ7f3bfArn66oKcYz5J0xAGEKMA7JBGa6wzI7aOTwdHbPofuhfOY+LpjTSUk)}}
> {{ ...}}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)