You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "201811510411lw (via GitHub)" <gi...@apache.org> on 2026/08/26 10:01:20 UTC

[I] [Bug][Connector-V2][MySQL-CDC] snapshotSplitColumn rejects a physical primary key [seatunnel]

201811510411lw opened a new issue, #11973:
URL: https://github.com/apache/seatunnel/issues/11973

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   When `snapshotSplitColumn` is explicitly configured with a physical primary key,
   SeaTunnel rejects it if the column is not returned by
   `JdbcDataSourceDialect.getUniqueKeys()`.
   
   Example table:
   
   ```sql
   CREATE TABLE test_split_column (
       id BIGINT NOT NULL AUTO_INCREMENT,
       category TINYINT NOT NULL,
       business_key VARCHAR(64) NOT NULL,
       PRIMARY KEY (id),
       UNIQUE KEY uk_business (category, business_key)
   );
   
   ### SeaTunnel Version
   
   2.3.13-SNAPSHOT
   
   The same issue is present on the current dev branch at commit:
   1491a8e0ad6f0f77ed4853fd4db3d4ff1cdfbd2e
   
   ### SeaTunnel Config
   
   ```conf
   source {
       MySQL-CDC {
           url = "jdbc:mysql://<host>:3306/test"
           username = "******"
           password = "******"
   
           database-names = ["test"]
           table-names = ["test.test_split_column"]
   
           table-names-config = [{
               table = "test.test_split_column"
               primaryKeys = ["id"]
               snapshotSplitColumn = "id"
           }]
   
           startup.mode = "initial"
           snapshot.split.size = 4096
           snapshot.fetch.size = 512
       }
   }
   ```
   
   ### Running Command
   
   ```shell
   # The job was submitted through SeaTunnel on Flink 
   # Flink version: 1.18.1
   
   # Equivalent command for a local reproduction:
   ./bin/start-seatunnel-flink-15-connector-v2.sh \
     --config ./config/mysql-cdc-primary-key.conf
   ```
   
   ### Error Exception
   
   ```log
   Config snapshotSplitColumn not unique key for table test.test_split_column
   
   Splitting table test.test_split_column into chunks,
   split column: category, min: 1, max: 2, chunk size: 4096
   ```
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][Connector-V2][MySQL-CDC] snapshotSplitColumn rejects a physical primary key [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11973:
URL: https://github.com/apache/seatunnel/issues/11973#issuecomment-5438933393

   Thanks for the detailed downstream verification, and thanks again to @waterWang and @zhangshenghang for the quick follow-up PRs.
   
   I rechecked the current `apache/dev` head (`e6caf5a4b337`). The issue is still grounded there: `AbstractJdbcSourceChunkSplitter.getSplitColumn()` first validates an explicit `snapshotSplitColumn` only against `dialect.getUniqueKeys()`, and only later falls back to primary-key / unique-key discovery for automatic split-column selection. That means an explicitly configured physical primary key can still be rejected before the automatic fallback path is reached.
   
   Your August 27, 2026 verification result is useful because it shows the intended behavior directly:
   - the configured physical primary key was accepted;
   - the previous `Config snapshotSplitColumn not unique key` path disappeared;
   - the generated snapshot splits used the configured primary-key boundaries.
   
   At this point, the strongest current repair signal is #11979: it has your downstream validation and its current Build is green. #11977 is still open but its current Build is failing. Please keep the fix path converged on one implementation and one focused regression test, rather than landing two near-duplicate repairs.
   
   For review, the contract should stay narrow:
   1. accept an explicitly configured split column when it belongs to the physical primary key or a unique key;
   2. preserve the existing automatic fallback behavior when the config is absent or invalid;
   3. keep the regression coverage focused on this exact configured-primary-key path.
   
   Unless the same scenario still reproduces on current `dev` with the #11979 logic, the next maintainer action belongs on PR review rather than more issue-thread debugging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][Connector-V2][MySQL-CDC] snapshotSplitColumn rejects a physical primary key [seatunnel]

Posted by "201811510411lw (via GitHub)" <gi...@apache.org>.
201811510411lw commented on issue #11973:
URL: https://github.com/apache/seatunnel/issues/11973#issuecomment-5434896342

   Thanks @waterWang and @zhangshenghang for investigating this issue and providing fixes.
   
   - @waterWang proposed #11977.
   - @zhangshenghang proposed #11979.
   
   I verified the core fix from #11979 after cherry-picking it into our downstream branch based on SeaTunnel 2.3.13.
   
   The test configuration included:
   
   ```hocon
   table-names-config = [{
       table = "<redacted_table>"
       primaryKeys = ["ID"]
       snapshotSplitColumn = "ID"
   }]
   
   startup.mode = "initial"
   snapshot.split.size = 4096
   ```
   
   The configured physical primary key was accepted as the snapshot split column:
   
   ```text
   Chosen split column ID for table <redacted_table>
   
   Splitting table <redacted_table> into chunks,
   split column: ID,
   min: 1,
   max: 1280590,
   chunk size: 4096
   
   Split table <redacted_table> into 305 chunks.
   ```
   
   The generated snapshot splits used non-null `ID` boundaries. For example:
   
   ```text
   SnapshotSplit(
       tableId=<redacted_table>,
       splitKeyType=ROW<ID BIGINT>,
       splitStart=[1223656],
       splitEnd=[1227861]
   )
   ```
   
   The previous behavior no longer occurred:
   
   ```text
   Config snapshotSplitColumn not unique key
   ```
   
   SeaTunnel did not fall back to another unique-key column.
   
   Therefore, the fix from #11979 resolves the reported issue in our SeaTunnel 2.3.13-based runtime.
   
   Thank you both again for the quick response and contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug][Connector-V2][MySQL-CDC] snapshotSplitColumn rejects a physical primary key [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx closed issue #11973: [Bug][Connector-V2][MySQL-CDC] snapshotSplitColumn rejects a physical primary key
URL: https://github.com/apache/seatunnel/issues/11973


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org