You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhang-arvin (via GitHub)" <gi...@apache.org> on 2026/08/19 01:47:51 UTC
[PR] [Feature][Connector-CDC] Support composite primary key in MySQL CDC snapshot split [seatunnel]
zhang-arvin opened a new pull request, #11867:
URL: https://github.com/apache/seatunnel/pull/11867
## Purpose
Add support for composite primary key (multi-column) in MySQL CDC snapshot chunk splitting. Previously only single-column primary keys were supported.
## Changes
- `AbstractJdbcSourceChunkSplitter`: multi-column split path
- `MySqlUtils`: multi-column split query generation with lexicographic comparison
- `MySqlChunkSplitter`: multi-column chunk split implementation
Closes #11742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-CDC] Support composite primary key in MySQL CDC snapshot split [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11867:
URL: https://github.com/apache/seatunnel/pull/11867#issuecomment-6051667476
@DanielLeens Thanks for the detailed review — the SQL-shape findings are correct and I've fixed the highest-severity ones. Pushed head `9d9d4c6fa`.
**Issue 1 (High, OR-chain parenthesization)** — fixed. Both lexicographic builders now wrap the whole OR chain in outer parentheses (and only parenthesize clauses after the first, so the result is `(a > ? OR (a = ? AND b >= ?))` instead of the precedence-broken `(a > ?) OR (...) AND ...`). Middle chunks can no longer leak rows past their upper bound.
**Issue 4 (param count/binding for N>=3)** — fixed. The placeholder count is now `N*(N+1)/2` (clause *i* has *i* equality values + 1 comparison) and `bindLexicographicLowerBoundParams` binds clause-by-clause instead of the wrong `2N-1` sequence.
**Issue 5 (tests could not detect the above)** — addressed. `MySqlUtilsTest` now asserts the **exact** generated SQL string (not substrings) for the middle/first/last split, plus a new 3-column middle-split case that locks the placeholder count. Full module suite green locally: `Tests run: 37, Failures: 0`.
**Issue 7 (unrelated tests)** — fixed. Removed the two snapshot-only startup tests (`MySqlIncrementalSourceStartupConfigTest`) that were carried into this PR but belong to the snapshot-only work (they also referenced a helper not present on this branch and broke the unit-test job).
Still open and not yet addressed in this push — I'd rather be explicit than claim otherwise: **Issue 2** (`getSplitColumns()` bypassing split-column selection semantics), **Issue 3** (binlog backfill boundary truncating composite keys to the first column), **Issue 6** (docs). I'll follow up on those next; flagging so you don't have to re-derive them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-CDC] Support composite primary key in MySQL CDC snapshot split [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #11867:
URL: https://github.com/apache/seatunnel/pull/11867#issuecomment-5352192504
@DanielLeens Thanks for the thorough review! I have addressed the key issues:
1. **Issue 1 (composite-key split path dead code)**: Added `getSplitColumns()` override in `MySqlChunkSplitter` and wired up the multi-column hooks (`queryMinMaxMulti`, `queryNextChunkMaxMulti`, `queryMinMulti`, `splitUnevenlySizedChunksMulti`).
2. **Issue 2 (parameter binding count bug for 3+ columns)**: Fixed the lexicographic placeholder-count and parameter-binding logic in `MySqlUtils`. The count formula now correctly derives from the same loop structure used to build the SQL string, and added unit tests for N=2, 3, and 4 columns.
3. **Issue 3 (getSplitType widening)**: Removed unused dead-code helpers (`getSplitColumns(Table)`, `getSplitType(Table)` overloads).
4. **Issue 5 (test coverage)**: Added unit tests for the composite key split scan query with multiple column counts.
Please take another look when you have a chance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org