You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "qidian99 (via GitHub)" <gi...@apache.org> on 2023/05/19 19:23:39 UTC

[GitHub] [incubator-paimon] qidian99 opened a new issue, #1181: [Feature] Support sync newly added table in CDC ingestion

qidian99 opened a new issue, #1181:
URL: https://github.com/apache/incubator-paimon/issues/1181

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   After CdcSyncDatabaseAction job start (for now we only sync a single database), if new table is created in MySQL, then action should sync the new mysql table to paimon. Specifically,
   
   1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix.
   2. CDC changelog of the newly added table should be successfully written to Paimon
   3. Support multiple newly added tables.
   
   
   ### Solution
   
   The draft design is as follows:
   1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix.
   
   Therefore we need to parse create table statement and utilize Paimon catalog to create the physical table
   
   2. CDC changelog of the newly added table should be successfully written to Paimon
   
   So we need another type of Paimon sink designed to write changelog from newly added tables to their corresponding Paimon tables. 
   
   3. Support multiple newly added tables.
   
   Specifically, the new Paimon sink needs to write the data from newly added tables discovered during runtime by managing multiple Paimon tables at the same time.
   
   ### Anything else?
   
   To accomplish this feature, it's better to go by several steps:
   
   1. PoC: implement a first draft to sync newly added table. 
   2. Implement a multiplexed Paimon sink to support multi-table write
   3. Implement a multiplexed Paimon committer to support multi-table commit
   4. End-to-end test against newly added table scenario
   
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] qidian99 commented on issue #1181: [Feature] Support sync newly added table in CDC ingestion

Posted by "qidian99 (via GitHub)" <gi...@apache.org>.
qidian99 commented on issue #1181:
URL: https://github.com/apache/incubator-paimon/issues/1181#issuecomment-1555130778

   There is a real-world use case as mentioned in #1106, so prioritize working on this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org