[jira] [Closed] (FLINK-39660) Kinesis Connector - Cascading failure in EFO subscription lifecycle

["Ferenc Csaky (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ferenc Csaky closed FLINK-39660.
--------------------------------
    Resolution: Fixed

[{{ede02da}}|https://github.com/apache/flink-connector-aws/commit/ede02dae4f0f98efe8bc174872f94db8194442b1] in main

> Kinesis Connector - Cascading failure in EFO subscription lifecycle
> -------------------------------------------------------------------
>
>                 Key: FLINK-39660
>                 URL: https://issues.apache.org/jira/browse/FLINK-39660
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kinesis
>    Affects Versions: aws-connector-5.0.0, aws-connector-6.0.0
>            Reporter: Chenyuan Lee
>            Assignee: Chenyuan Lee
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: aws-connector-6.1.0
>
>
> h2. Overview
> Under certain conditions, the Kinesis EFO subscription lifecycle in {{FanOutKinesisShardSubscription}} enters a cascading failure: an initial disruption in one subscription propagates into repeated subscription failures across many shards, causing the connector to stall.
> h2. Observed Symptoms
>  * {{ClosedChannelException}} bursts affecting many shards simultaneously.
>  * {{AcquireTimeoutException: Acquire operation took longer than 60000 milliseconds}} errors across many shards.
>  * Same shard retried from the same {{startingMarker}} for hundreds of iterations without forward progress.
>  * {{Http2PingHandler}} logs {{PING timer scheduled after N ms}} warnings clustered across many channels under load, indicating Netty event loop blocking during record processing.
>  * HTTP/2 connection exhaustion: SDK pool fills with pending {{subscribeToShard}} calls that never complete, each consuming a slot until individual 60-second acquire timeouts trigger.
> h2. Logs
> Example stack trace from an EFO subscription failure under cascading failure conditions:
> {noformat}
> Error onError subscribing to shard shardId-000000000144 with starting position ...
> java.io.IOException: An error occurred on the connection: java.nio.channels.ClosedChannelException, [channel: 76d2a9d0]. All streams will be closed
>     at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.decorateConnectionException(MultiplexedChannelRecord.java:213)
>     at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$closeChildChannels$10(MultiplexedChannelRecord.java:205)
>     at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$closeAndExecuteOnChildChannels$11(MultiplexedChannelRecord.java:229)
>     at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:259)
> {noformat}
> h2. Impact
> Multiple shards stop making progress; consumption lag grows and the condition can persist without manual intervention.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)