You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "huyuanfeng2018 (via GitHub)" <gi...@apache.org> on 2025/07/24 10:05:38 UTC

[PR] [cdc] Optimize SyncDatabaseAction performance by removing listTables calls [paimon]

huyuanfeng2018 opened a new pull request, #5956:
URL: https://github.com/apache/paimon/pull/5956

   ## Purpose
   from #5955 
   
   ### What is the purpose of the change
   Optimize SyncDatabaseAction performance by removing expensive listTables operations during initialization, improving scalability for databases with many tables.
   
   ### Brief change log
   - Remove `listTables()` call from `RichCdcMultiplexRecordEventParser`
   - Implement lazy table creation in `CdcDynamicTableParsingProcessFunction#processElement`
   - Remove `createdTables` Set to reduce memory usage
   
   ### Verifying this change
   - Verified existing functionality remains intact
   
   ### Testing
   This optimization does not require additional test cases as the existing functionality is already covered by:
   - `SyncDatabaseActionBaseTest.testSyncTablesWithoutDbLists()` - validates table filtering logic
   - `SyncDatabaseActionBaseTest.testSyncTablesWithDbList()` - validates database filtering logic  
   - `SyncDatabaseActionBaseTest.testSycTablesCrossDB()` - validates cross-database filtering scenarios
   
   All these tests create and use RichCdcMultiplexRecordEventParser, ensuring the optimization doesn't break existing functionality.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [cdc] Optimize SyncDatabaseAction performance by removing listTables calls [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #5956:
URL: https://github.com/apache/paimon/pull/5956


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [cdc] Optimize SyncDatabaseAction performance by removing listTables calls [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #5956:
URL: https://github.com/apache/paimon/pull/5956#issuecomment-4527157591

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [cdc] Optimize SyncDatabaseAction performance by removing listTables calls [paimon]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5956:
URL: https://github.com/apache/paimon/pull/5956#issuecomment-4414038136

   This pull request has had no activity for 90 days. If you'd like to keep it open, please push a new commit or leave a comment. Thanks for the contribution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org