You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "huyuanfeng2018 (via GitHub)" <gi...@apache.org> on 2025/07/24 09:49:30 UTC
[I] [Feature] Optimize CDC sync database action to avoid blocking on listTables operation with large number of tables [paimon]
huyuanfeng2018 opened a new issue, #5955:
URL: https://github.com/apache/paimon/issues/5955
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
When using CDC synchronous database operations on a database containing a large number of tables (hundreds of thousands), the "catalog. listTables (database)" operation in "SyncDataActionBase" may take a long time to complete, causing the entire synchronization job to start blocking for a long time. This will significantly affect the duration of CDC synchronization task initiation.
### Solution
### Current Behavior
The current implementation calls `catalog.listTables(database)` during initialization and maintains a `createdTables` set to track table creation status. This approach:
1. Blocks the entire sync process while listing all tables
2. Consumes unnecessary memory to maintain the `createdTables` set
3. Performs redundant operations when tables are created lazily
### Expected Behavior
The sync process should:
1. Avoid blocking on `listTables` operation during initialization
2. Create tables lazily when needed without maintaining a global `createdTables` set
3. Improve overall performance for databases with large numbers of tables
### Solution
Optimize the table creation logic by:
1. Removing the upfront `listTables` call in `SyncDatabaseActionBase`
2. Eliminating the `createdTables` set from `RichCdcMultiplexRecordEventParser`
3. Implementing lazy table creation in `CdcDynamicTableParsingProcessFunction` with existence checks
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Optimize CDC sync database action to avoid blocking on listTables operation with large number of tables [paimon]
Posted by "water-flower30 (via GitHub)" <gi...@apache.org>.
water-flower30 commented on issue #5955:
URL: https://github.com/apache/paimon/issues/5955#issuecomment-3567425588
Hello, I noticed that this issue has been paused for some time. Now we are also being affected by this issue. Can I proceed with the merging of this pr @huyuanfeng2018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Optimize CDC sync database action to avoid blocking on listTables operation with large number of tables [paimon]
Posted by "huyuanfeng2018 (via GitHub)" <gi...@apache.org>.
huyuanfeng2018 commented on issue #5955:
URL: https://github.com/apache/paimon/issues/5955#issuecomment-3573462376
> Hello, I noticed that this issue has been paused for some time. Now we are also being affected by this issue. Can I proceed with the merging of this pr [@huyuanfeng2018](https://github.com/huyuanfeng2018)
@yuzelin cc, PTAL～
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org