[jira] [Updated] (FLINK-40732) Release finished snapshot split metadata in the MongoDB CDC connector

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-40732:
-----------------------------------
    Labels: pull-request-available  (was: )

> Release finished snapshot split metadata in the MongoDB CDC connector
> ---------------------------------------------------------------------
>
>                 Key: FLINK-40732
>                 URL: https://issues.apache.org/jira/browse/FLINK-40732
>             Project: Flink
>          Issue Type: Improvement
>          Components: Flink CDC
>            Reporter: Spoorthi Basu
>            Priority: Major
>              Labels: pull-request-available
>
> The MongoDB CDC connector keeps the snapshot split metadata (assigned splits, finished offsets and table schemas) in the source coordinator for the whole lifetime of the job, and rebuilds it on every checkpoint. On a job with a very large number of snapshot splits this keeps JobManager memory high and inflates checkpoint size.
> That metadata is released by the incremental source framework ({{flink-cdc-base}}) once the source enters the stream phase, behind {{scan.incremental.snapshot.metadata.release.enabled}} (FLINK-40697). The four JDBC dialects inherit the option from {{BaseSourceConfig}} and its validation from {{JdbcSourceConfig}}. MongoDB implements {{SourceConfig}} directly, so it inherits neither and MongoDB jobs cannot enable the option.
> This proposes adding it to the MongoDB connector: the config plumbing, the fail-fast against {{scan.newly-added-table.enabled}}, and the option itself via both the DataStream builder and Flink SQL.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)