[jira] [Updated] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-8542:
----------------------------------
    Labels: auto-deprioritized-major stale-minor  (was: auto-deprioritized-major)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.


> Do not indefinitely store closed shard's state in the FlinkKinesisConsumer
> --------------------------------------------------------------------------
>
>                 Key: FLINK-8542
>                 URL: https://issues.apache.org/jira/browse/FLINK-8542
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kinesis
>            Reporter: Tzu-Li (Gordon) Tai
>            Priority: Minor
>              Labels: auto-deprioritized-major, stale-minor
>
> See original discussion here: [https://github.com/apache/flink/pull/5337|https://github.com/apache/flink/pull/5337#issuecomment-362227711]
> Currently, the Kinesis consumer keeps a list of {{(StreamShardMetadata, SequenceNumber)}} as its state. That list also contains all shards that have been closed already, and is kept in the state indefinitely so that on restore, we know that a closed shard is already fully consumed,
> The downside of this, is that the state size of the Kinesis consumer can basically grow without bounds, as the consumed Kinesis streams are resharded and more and more closed shards are present.
> Some possible solutions have been discussed in the linked PR comments.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)