[I] [Bug] Incorrectly including tables matching excludingTablePattern in combined mode cdc [paimon]

["chjnxp (via GitHub)" <gi...@apache.org>]

chjnxp opened a new issue, #4163:
URL: https://github.com/apache/paimon/issues/4163

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   master
   
   ### Compute Engine
   
   flink
   
   ### Minimal reproduce step
   
   1. Create a database named cdc_test in mysql, then create primary table named pk_1, pk_2 ... pk_100 and non-primary table named non_pk_1, non_pk_2, ... non_pk_100.
   2. Start a combined mode mysql database cdc job, and set excludingTablePattern to 'non_pk_.+'.
   3. Concurently create non-primary table named non_pk_101 when starting cdc job.
   4. Finally, the Jobmanager log will show "com.ververica.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including ‘cdc_test.non_pk_101’ for further processing".
   
   ### What doesn't meet your expectations?
   
   non_pk_101 obviously matches the excludingTablePattern and needs to be excluded. Since non_pk_101 is a non-primary key table, MySqlChunkSplitter will report an error: 
   Caused by: org.apache.flink.table.api.ValidationException: Incremental snapshot for tables requires primary key, but table cdc_test.non_pk_101 doesn't have primary key.
   
   ![image](https://github.com/user-attachments/assets/f5b72900-a8f7-44da-919b-316ee7e2d540)
   combinedModeTableList func uses excluding pattern ?!(^db\\.tbl$)|(^...$)，and this will miss the newly table that matches excludingTablePattern and created when starting cdc job.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Incorrectly including tables matching excludingTablePattern in combined mode cdc [paimon]

["1fanwang (via GitHub)" <gi...@apache.org>]

1fanwang commented on issue #4163:
URL: https://github.com/apache/paimon/issues/4163#issuecomment-5693406760

   I'd like to take this. In combined mode, a table created during startup can bypass `--excluding-tables`. I'll keep the exclusion pattern in source discovery and add a MySQL/Flink regression. Could you assign this to me?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org