You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "luozihen (via GitHub)" <gi...@apache.org> on 2026/08/31 09:15:00 UTC
[PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
luozihen opened a new pull request, #12015:
URL: https://github.com/apache/seatunnel/pull/12015
Purpose of this pull request
Close #11878.
Add a new JDBC sink option `multi-table_config.primary_keys` to support per-table
primary key mapping in multi-table generated-SQL scenarios.
When the upstream table name matches one of the configured patterns, the mapped key
columns are used; otherwise it falls back to the existing `primary_keys` / catalog
metadata logic. Inside this new option, `${primary_key}` / `${unique_key}` can be
mixed with static columns. The legacy top-level `primary_keys` contract is kept
unchanged.
This PR also fixes the config-parsing issue that blocked the feature: SeaTunnel
reconstructs config via `ConfigFactory.parseMap(...)`, which interprets map keys as
Typesafe Config path expressions. Regex keys such as `^t_nova_.*$` contain `$` and
`.`, so they fail with `ConfigException$BadPath`. The reconstruction now uses a JSON
round-trip (`ConfigFactory.parseString(..., JSON)`) in
`ConfigShadeUtils.processConfig` and `ReadonlyConfig.toConfig`.
### Does this PR introduce _any_ user-facing change?
Yes. Adds a new optional JDBC sink option `multi-table_config`.
Semantics:
- Each key is a Java regular expression matched against the upstream table name with
full-match semantics (`tableName.matches(pattern)`).
- Each value is a list of key columns. `${primary_key}` and `${unique_key}` are
supported only inside this option and can be mixed with static columns.
- Precedence: matched mapping wins; otherwise falls back to the existing logic
(top-level `primary_keys` -> catalog primary key -> first unique key -> plain INSERT).
- If a table matches multiple patterns, the first pattern in declaration order wins.
- If a matched table uses `${primary_key}` / `${unique_key}` but the upstream table has
no primary/unique key, the job fails with a clear error.
No existing option is renamed, removed, or changed in default behavior.
### Example configurations
#### Example 1
```hocon
env {
job.mode = "STREAMING"
parallelism = 1
checkpoint.interval = 10000
}
source {
MySQL-CDC {
plugin_output = "sharding00_source"
url = "jdbc:mysql://<host>:3306/st_source"
username = "<username>"
password = "<password>"
database-names = ["st_source"]
table-pattern = "st_source\\.t_(nova_(fo_serial|order)_(000|048)|tyuen_txn_(cp|qr)_(000|048))"
startup.mode = "initial"
server-id = "5400-5408"
}
}
transform {
Sql {
plugin_input = "sharding00_source"
plugin_output = "sharding00_transform"
query = "SELECT *, 'idc' AS DATA_SOURCE FROM dual"
}
}
sink {
Jdbc {
plugin_input = "sharding00_transform"
url = "jdbc:mysql://<host>:3306/st_target?rewriteBatchedStatements=true&autoReconnect=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "<username>"
password = "<password>"
generate_sink_sql = true
database = "st_target"
multi-table_config {
primary_keys {
"^t_nova_.*$" = ["${primary_key}", "DATA_SOURCE"]
"^t_tyuen_txn_.*$" = ["id_txn_ctrl", "DATA_SOURCE"]
}
}
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}
```
#### Example 2
```hocon
env {
job.mode = "STREAMING"
parallelism = 1
checkpoint.interval = 10000
}
source {
MySQL-CDC {
plugin_output = "gote_source"
url = "jdbc:mysql://<host>:3306/st_source"
username = "<username>"
password = "<password>"
database-names = ["st_source"]
table-pattern = "st_source\\.t_(nova_merchant_info|tyuen_txn_ext|nova_merge_settle_serial)"
startup.mode = "initial"
server-id = "5500-5508"
}
}
transform {
Sql {
plugin_input = "gote_source"
plugin_output = "gote_transform"
query = "SELECT *, 'idc' AS DATA_SOURCE FROM dual"
table_transform = [
{
table_path = "st_source.t_nova_merchant_info"
query = "SELECT * FROM dual"
}
]
}
}
sink {
Jdbc {
plugin_input = "gote_transform"
url = "jdbc:mysql://<host>:3306/st_target?rewriteBatchedStatements=true&autoReconnect=true"
driver = "com.mysql.cj.jdbc.Driver"
username = "<username>"
password = "<password>"
generate_sink_sql = true
database = "st_target"
primary_keys = ["merchant_id"]
multi-table_config {
primary_keys {
"t_tyuen_txn_ext.*" = ["id_txn_ctrl", "DATA_SOURCE"]
"t_nova_merge_settle_serial" = ["${primary_key}", "DATA_SOURCE"]
}
}
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}
```
### 建表 SQL
源表和目标表均为手动创建。`multi-table_config.primary_keys` 决定写入目标表时使用的主键列
(体现为最终生成的 upsert / delete 语句的 key 列)。
#### Demo 1 源表
```sql
CREATE TABLE t_nova_fo_serial_000 (
serial_id BIGINT NOT NULL,
order_no VARCHAR(64),
amount DECIMAL(18,2),
status VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (serial_id)
);
CREATE TABLE t_nova_order_000 (
id BIGINT NOT NULL,
order_no VARCHAR(64),
amount DECIMAL(18,2),
created_at DATETIME,
PRIMARY KEY (id)
);
CREATE TABLE t_tyuen_txn_cp_000 (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
amount DECIMAL(18,2),
channel VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (id),
UNIQUE KEY uk_id_txn_ctrl (id_txn_ctrl)
);
CREATE TABLE t_tyuen_txn_qr_000 (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
qr_code VARCHAR(128),
amount DECIMAL(18,2),
created_at DATETIME,
PRIMARY KEY (id),
UNIQUE KEY uk_id_txn_ctrl (id_txn_ctrl)
);
```
#### Demo 1 目标表
```sql
CREATE TABLE t_nova_fo_serial_000 (
serial_id BIGINT NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
order_no VARCHAR(64),
amount DECIMAL(18,2),
status VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (serial_id, DATA_SOURCE)
);
CREATE TABLE t_nova_order_000 (
id BIGINT NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
order_no VARCHAR(64),
amount DECIMAL(18,2),
created_at DATETIME,
PRIMARY KEY (id, DATA_SOURCE)
);
CREATE TABLE t_tyuen_txn_cp_000 (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
amount DECIMAL(18,2),
channel VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (id_txn_ctrl, DATA_SOURCE)
);
CREATE TABLE t_tyuen_txn_qr_000 (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
qr_code VARCHAR(128),
amount DECIMAL(18,2),
created_at DATETIME,
PRIMARY KEY (id_txn_ctrl, DATA_SOURCE)
);
```
#### Demo 2 源表
```sql
CREATE TABLE t_nova_merchant_info (
id BIGINT NOT NULL,
merchant_id VARCHAR(64) NOT NULL,
merchant_name VARCHAR(128),
mcc VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (id),
UNIQUE KEY uk_merchant_id (merchant_id)
);
CREATE TABLE t_tyuen_txn_ext (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
txn_amount DECIMAL(18,2),
ext_info VARCHAR(255),
created_at DATETIME,
PRIMARY KEY (id),
UNIQUE KEY uk_id_txn_ctrl (id_txn_ctrl)
);
CREATE TABLE t_nova_merge_settle_serial (
id BIGINT NOT NULL,
settle_no VARCHAR(64),
amount DECIMAL(18,2),
settle_date DATE,
created_at DATETIME,
PRIMARY KEY (id)
);
```
#### Demo 2 目标表
```sql
CREATE TABLE t_nova_merchant_info (
id BIGINT NOT NULL,
merchant_id VARCHAR(64) NOT NULL,
merchant_name VARCHAR(128),
mcc VARCHAR(16),
created_at DATETIME,
PRIMARY KEY (merchant_id)
);
CREATE TABLE t_tyuen_txn_ext (
id BIGINT NOT NULL,
id_txn_ctrl VARCHAR(64) NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
txn_amount DECIMAL(18,2),
ext_info VARCHAR(255),
created_at DATETIME,
PRIMARY KEY (id_txn_ctrl, DATA_SOURCE)
);
CREATE TABLE t_nova_merge_settle_serial (
id BIGINT NOT NULL,
DATA_SOURCE VARCHAR(32) NOT NULL,
settle_no VARCHAR(64),
amount DECIMAL(18,2),
settle_date DATE,
created_at DATETIME,
PRIMARY KEY (id, DATA_SOURCE)
);
```
### 生成的 SQL 结果
下面是用上面两个 demo 跑通后,MySQL JDBC sink 实际生成的部分 upsert / delete 语句。
可以看出每个目标表已经按 `multi-table_config.primary_keys` 使用了不同的主键列。
```text
-- Example 1
INSERT INTO `st_target`.`t_nova_fo_serial_000`
(`serial_id`, `order_no`, `amount`, `status`, `created_at`, `DATA_SOURCE`)
VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ...;
DELETE FROM `st_target`.`t_nova_fo_serial_000`
WHERE `serial_id` = ? AND `DATA_SOURCE` = ?;
INSERT INTO `st_target`.`t_tyuen_txn_cp_000`
(`id`, `id_txn_ctrl`, `amount`, `channel`, `created_at`, `DATA_SOURCE`)
VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ...;
DELETE FROM `st_target`.`t_tyuen_txn_cp_000`
WHERE `id_txn_ctrl` = ? AND `DATA_SOURCE` = ?;
-- Example 2
INSERT INTO `st_target`.`t_nova_merchant_info`
(`id`, `merchant_id`, `merchant_name`, `mcc`, `created_at`)
VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ...;
DELETE FROM `st_target`.`t_nova_merchant_info`
WHERE `merchant_id` = ?;
INSERT INTO `st_target`.`t_tyuen_txn_ext`
(`id`, `id_txn_ctrl`, `txn_amount`, `ext_info`, `created_at`, `DATA_SOURCE`)
VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ...;
DELETE FROM `st_target`.`t_tyuen_txn_ext`
WHERE `id_txn_ctrl` = ? AND `DATA_SOURCE` = ?;
INSERT INTO `st_target`.`t_nova_merge_settle_serial`
(`id`, `settle_no`, `amount`, `settle_date`, `created_at`, `DATA_SOURCE`)
VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ...;
DELETE FROM `st_target`.`t_nova_merge_settle_serial`
WHERE `id` = ? AND `DATA_SOURCE` = ?;
```
### How was this patch tested?
Added unit tests:
- `JdbcSinkFactoryTest`: mapping hit, fallback to legacy `primary_keys`, missing upstream
primary key error, first-match-wins, string value handling, and the full
factory-context path.
- `ConfigShadeTest`: special-character config keys are preserved after `decryptConfig`.
- `ReadableConfigTest`: special-character config keys are preserved after
`ReadonlyConfig#toConfig`.
Manually verified with MySQL-CDC -> JDBC using the two demo configurations above
(`generate_sink_sql = true`), and confirmed the manually-created target tables together
with the generated upsert / delete statements match the expected per-table primary key
mapping.
Also run `./mvnw spotless:apply`.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5690923131
@luozihen Thanks for the F7 and F8 responses.
On F8, the snake_case rename to `multi_table_config` is the right direction. I haven't verified the rename in the diff myself yet, so I'll take a look at `JdbcSinkOptions` and the error strings, tests, and EN/ZH docs before marking it resolved.
On F7, `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` sounds like the regression guard I was after; I'll confirm in the diff that it asserts the old dotted-key expansion behaviour. Deferring the engine-level IT/E2E for `multi_table_config.primary_keys` to a Follow-up section in the PR description is fine with me. One small ask: once you open the separate tracking item for it, please link it from that section so it doesn't get lost after merge.
For F1-F6, I don't see responses from you in this thread yet. Could you post a short note per item describing how each was addressed, ideally pointing at the relevant code or test? F1 and F2 in particular I'd like to re-verify against the diff before closing.
<!-- streview-comment:1070 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5867647162
@luozihen thanks for pulling the logs and asking before burning more retries - please stop retrying, it will not help.
**This is deterministic, not flaky.** `Can't get Docker image ... quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z` means the MinIO test image that the S3-based ITs were pointing at is no longer pullable anonymously (Docker Hub `minio/minio` went away first, then the quay.io copy). Every run that reaches an S3/MinIO-based IT fails at container start, regardless of your code. Nothing in this PR touches those tests.
**It is already fixed on `dev`, and your branch does not have the fix yet.** I checked the current heads directly:
- `dev` no longer references `quay.io/minio/minio:RELEASE...` anywhere. The S3/MinIO ITs (S3FileWithFilterIT, the Hudi, Iceberg and Paimon S3 ITs, the Databend ITs and others) now use a digest-pinned public mirror of MinIO (10 files), which came in with #12297, and #12478 additionally waits for MinIO readiness in S3FileWithFilterIT.
- Your head (`00acc23a5`) is 60 commits behind `dev` and still has the old tag in 9 test files, so `all-connectors-it-2 (8/11)` will keep failing until the branch picks up those changes.
**Next step (lowest cost):** please sync your branch with the latest `dev` (merge or rebase, whichever you prefer) and push, then let CI run on the new head. Please do not edit any of the MinIO test files in this PR yourself - the sync brings the correct versions in. If `all-connectors-it-2` still fails after the sync, paste the new failing job link here and I will look at it against the updated head.
Your source-side review status is unchanged: I have no new code-side blocker on this PR, the remaining gate is CI on a synced head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5487038507
@DanielLeens
Hi, I accidentally merged the upstream dev branch into my dev branch before opening this PR (commit hash: [2acb55a048d0347c285319e39ef76cf55d252213]). This caused the CI tests to fail because of the upstream code, while my own changes are fully tested and working.
Since this merge commit has already been pushed, I'd like to ask for your preferred way to fix this:
Safe way (Revert): I can run git revert -m 1 [hash] and push normally. This will add a revert commit to cancel the upstream changes and pass the CI, but it leaves a revert commit in history.
Clean way (Rebase): I can drop that merge commit via interactive rebase and force push, but it will rewrite the remote history.
I'm leaning towards option 1 to avoid force-push issues. Please let me know your preference. Sorry for the noise!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5812758363
@luozihen thanks for pinning the F1-F8 checklist to concrete `path:line` locations for @SEZ9 - I independently spot-checked every citation against the current head (`00acc23a582d76a589c00df956dded4b6f21da94`) via `git show`/`git grep`, no trust-the-description shortcuts, and they all line up exactly:
- F1: `ReadonlyConfig.java:76` is `toConfig()` (unchanged); `ConfigShadeUtils.java:224` is the JSON-syntax reparse inside `processConfig` (the shade-rebuild fix); `MultiTableFailureHelper.java:64` is `mergeOptions(...)`.
- F2/F5: `JdbcSinkFactory.java:314` is `resolveMultiTablePrimaryKeys(...)`, `:344` is `toCompiledPatternMap(...)` (the `LinkedHashMap` that guarantees first-match-wins).
- F3: `:282` is `validatePrimaryKeyColumns(...)`.
- F4: `:416` is `expandPrimaryKeyPlaceholder(...)`.
- F6: `:361` is `compilePattern(...)`.
- F8: `JdbcSinkOptions.java:112` is the `MULTI_TABLE_CONFIG` option declaration - fully snake_case as claimed.
- F7: both `ReadableConfigTest` and `JdbcMysqlMultipleTablesIT` exist at the paths given in the PR description and cover what's claimed.
Since the head SHA hasn't changed since my last full re-review and approval, I don't have anything new to add here beyond confirming these line citations are accurate - this is just closing out @SEZ9's verification request, not a functional change. My "Ready to merge" conclusion from the last round still stands; still waiting on a green CI run and a write-capable maintainer for the final merge action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5707320566
@luozihen Thanks for moving the F1–F8 status into the PR description — that works better than the truncated comments, and I'll use the "Review status (F1–F8)" section as the source of truth on the next pass.
To make that pass quick, here's what I'll be checking against for each item, so please make sure the description points me at the concrete change (or states explicitly that you're pushing back and why):
- **F1 / F7** – whether the `ReadonlyConfig.toConfig()` dotted-key behavior is preserved for existing callers, and where the regression test for the old expansion lives (e.g. in `ReadableConfigTest`), plus which IT/E2E exercises the new JDBC option end to end.
- **F2 / F5** – how "first pattern in declaration order wins" is actually guaranteed given the `Map<String, Object>` option type, or that the docs in `docs/en/connectors/sink/Jdbc.md` were reworded to match the real behavior.
- **F3** – where the resolved key columns are validated as identifiers before they are joined into `PRIMARY_KEYS` and reach generated SQL.
- **F4** – a short description (ideally with a test) of the ordering between the connector-level `${primary_key}`/`${unique_key}` expansion and the engine-level `TablePlaceholder` replacement.
- **F6** – where user-supplied regexes are compiled up front and mapped to the JDBC-12 error code.
- **F8** – the final option key name (the `multi-table_config` hyphen/underscore mix), or the reasoning if you want to keep it.
One unrelated note on CI: the failures on run `34802808218` for head `2c5982be8` are `minio/minio` image pulls in Paimon/Databend/S3 tests, none of which are touched by this PR; a rebase onto `dev` should clear them. Once the branch is rebased and the description status is complete, ping me and I'll go through F1–F8 in one pass.
<!-- streview-comment:1096 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673716659
@SEZ9
F1 — `ReadonlyConfig#toConfig()` is unchanged: byte-for-byte identical to `dev`
(`ConfigFactory.parseMap(confData)`). The fix is scoped to
`MultiTableFailureHelper.mergeOptions()` (seatunnel-api), which now merges `getSourceMap()`
directly instead of round-tripping through `toConfig()`. I also added a Javadoc note there stating
the merge is shallow/top-level (primary replaces fallback's value wholesale on a key collision).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5524531439
@DanielLeens
Thanks for the thorough re-review and for laying out the three issues so clearly.
I've addressed all of them in the latest push:
1. **Scoped down the shared-API change (Issue 1)** — reverted `ReadonlyConfig#toConfig()` back to
`ConfigFactory.parseMap(confData)`, so the existing dotted-key expansion semantics are left
untouched. The JDBC feature no longer depends on changing `toConfig()`; instead,
`MultiTableFailureHelper.mergeOptions()` now merges the underlying option maps directly
(primary takes precedence), which avoids re-parsing the regex keys as HOCON paths.
2. **Extracted the duplicated PK-fallback logic (Issue 2)** — moved it into
`applyPrimaryKeys` / `applyFallbackPrimaryKeys`, called from both branches.
3. **Added Javadoc (Issue 3)** — documented the matching and placeholder-expansion contract for
`resolveMultiTablePrimaryKeys` and its private helpers.
I also updated the tests to match: removed the now-obsolete
`ReadableConfigTest#testToConfigPreservesSpecialCharacterKeys`, and added a
`MultiTableFailureHelperTest` case verifying that `mergeOptions` preserves the special-character
keys.
The changes are committed and CI is running on the new head. I'll ping here again once it finishes.
Regarding the previous CI run, I noticed an error that I wasn't able to interpret on my own:
```text
Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500:
{"message":"Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting
for connection (Client.Timeout exceeded while awaiting headers)"}
```
It looks like it timed out pulling an image from Docker Hub, so it may be a transient network
issue rather than a problem with the changes. I'm not confident about that conclusion though, so
if this (or anything similar) shows up again in the new run, I'd really appreciate a pointer on
how to interpret and troubleshoot it.
Thanks again for your patience and guidance throughout this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5617127743
@luozihen The bounded LRU is fine — it's exactly the fallback I offered as an alternative to per-job scoping ("if a cross-job cache is preferred for its simplicity, bound it ... rather than an unbounded ConcurrentHashMap"), so no need to hold off before I weigh in — go ahead and push it.
On per-job scoping: your read of the constraint is right, and I won't ask you to touch the engine's sink-planning path for this PR. `TableSinkFactory`/`createSink()` genuinely has no per-job hook to thread a pre-compiled pattern map through, and reworking that is out of scope for a connector-level change. The bounded LRU keeps the fix local, which is the right trade-off here.
On the snippet: the shape is correct. `new LinkedHashMap<>(16, 0.75f, true)` with the third argument `true` gives you access-order iteration, which is what actually makes `removeEldestEntry` evict the least-recently-used entry rather than the oldest-inserted one (leaving that flag off is a common mistake in hand-rolled LRUs) — capped at 256, wrapped in `Collections.synchronizedMap` for thread safety on individual `get`/`put` calls. The same benign check-then-act race as before still applies: two threads can both miss and both compile+cache the same pattern under load, but `Pattern.compile()` is deterministic, so the worst case is one redundant compile, never a correctness issue.
One thing worth a sentence in the field comment or PR description: 256 is a reasonable default for the common case (a handful of table-matching patterns reused across resubmissions), but if you have a sense of how many distinct patterns a busy multi-tenant deployment might realistically configure, it's worth noting that 256 is comfortably above that rather than picked arbitrarily. Not a blocker.
Push it and ping me once CI is green, and I'll take a final pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5630162368
@SEZ9 @DanielLeens
Good news — CI is green on the latest head.
Here's the per-item status:
- **F1** — Narrowed to the multi-table sink path. `ReadonlyConfig#toConfig()` is now byte-for-byte the
pre-PR version (unchanged); the config-reconstruction fix is scoped to
`MultiTableFailureHelper.mergeOptions()`, which merges the option maps directly instead of going
through `toConfig()`.
- **F2 / F5** — Chose the order-preserving structure. `toCompiledPatternMap` builds an explicit
`LinkedHashMap<Pattern, List<String>>`, so "first declared pattern wins" is guaranteed by
construction rather than by the source map's incidental order. `testResolveMultiTablePrimaryKeysFirstMatchWins`
covers it, and the statement in `docs/en/connectors/sink/Jdbc.md` matches this behavior.
- **F3** — Done. `validatePrimaryKeyColumns` (called from the shared `applyPrimaryKeys` helper)
rejects blank names and names containing a comma before they are comma-joined into `PRIMARY_KEYS`.
Covered by `testFactoryContextWithMultiTableConfigInvalidColumnFails`.
- **F4** — The connector expands `${primary_key}` / `${unique_key}` inside
`multi_table_config.primary_keys`; the engine-level
`TablePlaceholderProcessor.replaceTablePlaceholder` only rewrites top-level String and
single-element-String-List values, so it leaves the nested map untouched — no double processing.
- **F6** — Done. `toCompiledPatternMap` compiles every declared pattern up front via `compilePattern`
(memoized in a bounded LRU), so an invalid regex raises `JDBC-12` immediately. Covered by
`testResolveMultiTablePrimaryKeysInvalidRegexFails` and `testCompilePatternInvalidFails`.
- **F7** — Partial, as agreed. `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` is the
regression test for the old `toConfig()` dotted-key behavior. The engine-level IT/E2E is deferred
and noted as a follow-up in the PR description.
- **F8** — Done. The key is now `multi_table_config` (snake_case) consistently across code, error
strings, unit tests, and EN/ZH docs.
On the cache sizing: the keys are table-name regexes, so a busy multi-tenant deployment typically
configures on the order of tens of distinct patterns (a few table families); 256 sits comfortably
above that.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6228523437
@luozihen Thanks for the ping, and apologies for the delay.
I see CI run `38840923843` on head `231cbdea5` is reported green, so CI is no longer a gate from my side.
Before I give a clear yes on merging, I'd like to confirm the status of the two HIGH items from the earlier review on this head:
1. The `ReadonlyConfig.toConfig()` dotted-key round-trip change, which affects every caller rather than only the JDBC sink (and whether a regression test for the old behaviour exists).
2. The documented "first pattern in declaration order wins" guarantee, given the option is a map whose key order is not preserved.
Could you point me to where these were addressed in the current head, or say explicitly if you intend to leave them as-is? If they are resolved, I'm fine with your plan to merge first and treat whatever remains of the Medium items (compile-time regex validation mapped to the `JDBC-12` error code, identifier validation of the resolved key columns before they reach `PRIMARY_KEYS`/generated SQL, the `${primary_key}`/`${unique_key}` vs engine-level placeholder interaction, and the `multi-table_config` naming) as follow-ups, ideally tracked in a follow-up issue.
<!-- streview-comment:1809 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6410379045
Noting that #12443 has been approved (thanks @hutiefang76 and @DanielLeens).
To avoid two competing config conversions, I will hold further config-loading changes here until #12443 lands:
- once it is merged, I will sync `dev` and drop `0c43f10dd` (the `ConfigBuilder.of(Map)` commit) from this PR, then rebase;
- the JDBC feature and the primary-key validation stay in this PR unchanged;
- the REST-shaped regression (`map -> RestUtil.buildConfig -> ConfigShadeUtils.decryptConfig`) and the `toConfig()` assertions will be rebased on top of #12443's contract, so they pin the final behaviour instead of a competing one.
The head stays at `d9bb1574f` until then. No action needed from you — I will post the rebased head here once #12443 is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6473887447
Thanks @SEZ9, @nzw921rx and @luozihen. The sequencing is unchanged: #12443 remains open at 509fddabe2ca, so please land that config-key contract first, then sync this branch and remove its competing conversion while retaining the combined REST/toConfig regression. The unchanged head here does not resolve that dependency. I also respect the maintainer's after-3.1.0 scheduling note; there is no need for an empty trigger commit or unrelated patch while waiting for the prerequisite.
Gate snapshot (2026-10-28T15:57:29+00:00), head `d9bb1574f0dfee9f6673533dff2616522a76e6c4`: [Build](https://github.com/apache/seatunnel/runs/119635588155) is `FAILURE`.
The blocking jobs are [Run / Dead links](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119601643901) `failure`; [Run / unit-test (17, ubuntu-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422299) `failure`; [Run / unit-test (11, windows-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422330) `failure`; [Run / unit-test (17, windows-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422406) `failure`; [Run / unit-test (11, ubuntu-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422431) `failure`; [Run / mysql-cdc-connector-it (17, ubuntu-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422535) `cancelled`; [Run / mysql-cdc-connector-it (11, ubuntu-latest)](https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422594) `cancelled`; [Run / transform-v2-it-part-2 (17, ubuntu-latest)]
(https://github.com/luozihen/seatunnel/actions/runs/39710874854/job/119602422616) `failure`.
Observed log evidence: `Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project seatunnel-transforms-v2: There are test failures.`
Please inspect or rerun only the named failed job where a rerun is appropriate. The logs establish the failing stage, but do not alone establish whether the current PR caused it. No matching upstream cure has been proved by this follow-up; a base sync alone is not a verified resolution, so no unrelated module/dependency patch is requested.
My [previous source conclusion](https://github.com/apache/seatunnel/pull/12015#issuecomment-6414884620) remains unchanged, including its independent source/documentation requests. CI is not being passed by inference, and this follow-up is not an effective approval.
Comparison recorded in this scan: base `dev`, `diverged`, ahead 33 / behind 74, no merge conflict. Branch drift is metadata here and is not an additional sync request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5611887428
@DanielLeens thanks for re-reviewing the whole feature end to end on `dcae482241c` rather than only the `608b46ba87` delta on top of `1730aeef` — much appreciated.
On the two items your comment covers:
- **F8 (`multi-table_config` -> `multi_table_config`)**: thanks for checking that no stale `multi-table_config` occurrences remain across the code, error message, docs and tests, and that the name now matches `primary_keys` / `generate_sink_sql`. I'll confirm against the diff on my side before marking it resolved.
- **F3 (identifier validation before comma-join)**: thanks for tracing that `validatePrimaryKeyColumns` is called from the shared `applyPrimaryKeys` helper, so the pre-existing top-level `primary_keys` path via `applyFallbackPrimaryKeys` gets the same fail-fast `JDBC-12` behavior — that is exactly the point I wanted covered. Same here: I'll verify it in the diff before closing.
Your comment appears to be cut off after the F3 section, so I can't see your conclusions on the remaining points. Could you add a short status on each?
1. **F2 / F5 (pattern precedence)**: does `608b46ba87` actually guarantee "first pattern in declaration order wins" for `multi_table_config.primary_keys`, or were the docs adjusted to describe the real behavior instead?
2. **F6 (regex validation -> `JDBC-12`)**: where are invalid user-supplied regexes compiled and rejected, and is there a test for it?
3. **F7 (tests)**: is there now a regression test for the old `toConfig()` dotted-key expansion, and does any engine-level IT/E2E exercise the new option?
4. **F1 (`toConfig()` dotted-key semantics) and F4 (`${primary_key}`/`${unique_key}` vs engine-level placeholder replacement)**: a one-line status on each from your re-trace would be enough.
Once those are covered and I've checked the diff, I'm happy to sign off on the current head.
<!-- streview-comment:939 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673726664
@SEZ9 F5 — The doc statement matches the code: `docs/en/connectors/sink/Jdbc.md:315` says "the first
pattern in declaration order wins", which is exactly what `toCompiledPatternMap`'s `LinkedHashMap`
guarantees and what `testResolveMultiTablePrimaryKeysFirstMatchWins` asserts.
`docs/zh/connectors/sink/Jdbc.md:315` carries the equivalent sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673565372
Thanks for re-posting, @luozihen — the tail of the previous message was indeed cut off.
**F4** — Thanks for laying out the ordering: `TablePlaceholderProcessor.replaceTablePlaceholder` running first but only touching top-level String / single-element-String-List values, leaving the nested `multi_table_config` map for `expandPrimaryKeyPlaceholder`, and failing fast with `JDBC-12` when `${primary_key}` is used against a table without a primary key. That is the behavior I was hoping for. Could you point me to the changed lines in this PR that implement the fail-fast path (and any test that exercises it) so I can confirm against the diff? It would also help to state the fail-fast semantics in the option docs so users know what to expect when a matched table has no PK.
**F6** — Compiling every pattern up front in `toCompiledPatternMap` via `compilePattern` and surfacing `JDBC-12` immediately is what I was after. Please point me to `testResolveMultiTablePrimaryKeysInvalidRegexFails` and `testCompilePatternInvalidFails` in the diff and I will mark this resolved.
**F7** — Good to hear `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` is in place; I will verify it in the changed test file. Your message was truncated again at "noted as a follow-up in the P" — could you confirm where the deferred engine-level IT/E2E is tracked (PR description or a tracking issue) and paste the reference here? I am fine deferring the E2E as long as it is explicitly tracked.
Still open from the previous review: F1, F2, F3, F5 and F8. If any of those were also in the truncated part of your reply, please re-post them the same way.
<!-- streview-comment:1045 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5754813722
Thanks for pulling the full log and laying out the evidence — that made this quick to follow up on.
**`JdbcMysqlMultipleTablesIT` failure (relates to PR12015-F7)**
In fork run 35303656875 (head `c15f9fdc5`) the failure is `All parts of a PRIMARY KEY must be NOT NULL`: the MySQL DDL builder emits an explicit `NULL` for nullable columns and then appends `PRIMARY KEY (...)` when `create_index` is true. That is a pre-existing gap that the top-level `primary_keys` path hits as well, so it is not introduced by this PR — but it is triggered by this PR's new test case.
To keep the PR scoped:
1. Please do not change `MysqlCreateTableSqlBuilder` (or the OceanBase MySQL-mode twin) here. Open a separate issue for the "emit `NOT NULL` for primary-key columns" fix; a follow-up PR with a unit test would be very welcome.
2. Fix the E2E fixture instead: add two small dedicated source tables for this scenario whose key columns (e.g. `c_int`, `c_integer`, `c_mediumint`) are declared `NOT NULL`, and point the patterns in the new `.conf` at them. Please leave the shared `source.table1` / `source.table2` definitions untouched, since the other test methods depend on them. Your assertions on resolved primary keys and row counts can stay as they are.
Once that is in, the IT/E2E half of F7 is covered. The other half — a regression test for the previous `toConfig()` dotted-key expansion behaviour — is still open.
**Option name (PR12015-F8)**
The thread now refers to the option as `multi_table_config`, while my earlier note was on `multi-table_config`. I'll confirm the key in `JdbcSinkOptions` and in the docs against the diff on the next pass; if both consistently use `multi_table_config`, F8 is resolved.
**Still open from the previous review**
I haven't seen responses yet on F1 (`ReadonlyConfig.toConfig()` JSON round-trip changing dotted-key semantics for all callers), F2/F5 (declaration-order guarantee for a `Map<String, Object>`-typed option and the matching docs claim), F3 (identifier validation before the resolved keys are joined into `PRIMARY_KEYS`), F4 (interaction with engine-level `${primary_key}`/`${unique_key}` placeholder replacement) and F6 (compile-time regex validation mapped to the JDBC-12 error code). A short note on how you plan to handle each would help me do the next pass in one go.
<!-- streview-comment:1207 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6331227461
Thanks @nzw921rx for re-checking the head so carefully — both points were real, and both are fixed now.
1. Empty / malformed key columns. toPrimaryKeyList() now receives the pattern and rejects, with JDBC-12 naming the pattern: list elements that are not strings (including null, which used to be stringified into the column name "null"), and values that produce no column at all ([], "", " "). The per-table blank/comma checks in validatePrimaryKeyColumns() remain as the final guard, so [""] still fails with JDBC-12. New tests in JdbcSinkFactoryTest cover [] (both resolveMultiTablePrimaryKeys() and the createSink path that previously reached JdbcSink#createWriter()), "", [""], [null], [123], a non-list/non-string value, a null value, and a positive comma-separated string, asserting the JDBC-12 code. The negative tests fail with "nothing was thrown" against the previous implementation.
2. REST JSON body. Fixed here as well. ConfigBuilder.of(Map) — used by RestUtil.buildConfig() and buildConfigList() — now rebuilds with the same literal-key helper as the config-file and shade paths (ConfigBuilder#rebuildWithLiteralKeys) instead of ConfigFactory.parseMap, keeping the resolve / resolveWith(systemProperties) steps. ConfigBuilderTest#testOfMapPreservesLiteralRegexOptionKeys runs the exact REST sequence (map → of(Map) → ConfigShadeUtils.decryptConfig) and asserts the regex key and its values survive; it fails with ConfigException.BadPath before the change. The EN/ZH docs now state that regex keys work for config files and for both the JSON and HOCON REST bodies.
On #12434 / #12443. With this change the REST JSON conversion is covered here, so #12434 is addressed by this PR. #12443 targets the same ConfigBuilder.of(Map) / ConfigShadeUtils conversion, so it would duplicate this fix and conflict in ConfigBuilder; it may be better to close it or put it on hold until this PR lands. If the maintainers would rather keep #12443 as the single REST fix, I can drop the REST commit from this PR instead — just tell me which way you prefer.
CI is queued on the new head d9bb1574f (which includes a fresh merge of dev); I will post the result once it finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6379498979
Thanks for the follow-up. The head is still `d9bb1574f0dfee9f6673533dff2616522a76e6c4`, so my latest review stands; this is a clarification on PR12015-F1 (and the related test gap in PR12015-F7), not a new review.
**On the semantics table (F1).** The first row says `job.mode` / `shade.identifier` are kept as literal keys when the config is built from a map. My reading of the shaded config parser points the other way: keys supplied via a map are parsed as paths and split on `.`, including keys inside nested maps such as `env`. That is the same mechanism that makes a regex key like `^t_nova_.*$` fail with `BadPath`. I could not find a change in this PR that alters that parsing, so unless you can point me at the exact class that does, I have to treat expansion as the current behavior, which means the literal rebuild in `0c43f10dd` could stop an `env`-supplied `shade.identifier` from being resolved the way it is today.
This is easiest to settle with a test rather than discussion, and you already proposed the right one:
1. Please add the REST-shaped test you described (map -> `RestUtil.buildConfig` -> `ConfigShadeUtils.decryptConfig`) with `env` containing `shade.identifier`, `job.mode`, `a.b` and one regex key, asserting which keys expand and which stay literal on the literal rebuild. If it is green I will withdraw F1 and say so here; if it is red, it tells us which direction to take.
2. Either way, please correct or confirm the first row of the semantics table so the doc and the test agree.
3. If the test is red, I'd lean toward the narrow option: keep every key that parses as a path today behaving exactly as it does now, and only rescue keys that fail path parsing. That keeps the change contained to the JDBC sink use case instead of altering `toConfig()` for every caller.
Once the test and the table update are pushed, I'll re-check F1 and F7 against the new head.
<!-- streview-comment:1995 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5577495364
@DanielLeens Thank you for your guidance. The CI checks have now turned green. Could you please advise on the next steps? This is my first time submitting a PR, so I’m not entirely familiar with the full process. Apologies for any inconvenience, and thank you for your time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673733840
@SEZ9 F8 — Confirmed snake_case: the option key is `multi_table_config` in `JdbcSinkOptions`
(`JdbcSinkOptions.java:113`), and in all error strings, unit tests, and EN/ZH docs. No
`multi-table_config` spelling remains anywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5749193893
Hi @luozihen, thank you for pulling the full log and for laying the evidence out so clearly. That made this quick to confirm. I re-checked it against the log of the fork run you linked and against `dev`.
**Confirmed**
- In fork run 35303656875 (head `c15f9fdc5`), `JdbcMysqlMultipleTablesIT.testMysqlJdbcMultipleTableE2e` fails on all 7 containers (`Tests run: 28, Failures: 7`), always at the `executeJob("/jdbc_mysql_source_and_sink_with_multi_table_config.conf")` call, with `SQLSyntaxErrorException: All parts of a PRIMARY KEY must be NOT NULL` wrapped in `CatalogException ... Failed creating table sink.table1`.
- Your diagnosis holds. `MysqlCreateTableSqlBuilder.java:205-209` on `dev` emits an explicit `NULL` for every nullable column, and `:166` appends `PRIMARY KEY (...)` when `create_index` is true, so a key on a nullable column is rejected by MySQL.
- It is pre-existing and not specific to `multi_table_config`. The existing top-level `primary_keys` option goes through the same path (`JdbcSinkFactory.java:146-155` on `dev` builds a `PrimaryKey` over the source columns without touching their nullability), and `OceanBaseMysqlCreateTableSqlBuilder.java:205` has the same code. I searched issues and PRs in this repo and could not find an existing report for it.
**My suggestion: option 2, keep this PR scoped**
1. Do not change `MysqlCreateTableSqlBuilder` here. It is a shared DDL builder used by every MySQL sink with `create_index = true` (and the OceanBase MySQL-mode twin has the same code), so changing it deserves its own review and tests, and it is easier for reviewers to reason about `multi_table_config` when this diff stays focused on the new option.
2. Make the E2E fixture valid for the scenario. Please avoid altering the shared `source.table1` / `source.table2` columns, because the other test methods in `JdbcMysqlMultipleTablesIT` depend on them. Creating two small dedicated source tables for this phase, whose key columns (for example `c_int`, `c_integer` and `c_mediumint`) are declared `NOT NULL`, and pointing the patterns in the new `.conf` at those names, keeps your assertions on the resolved primary keys and row counts exactly as they are.
3. Please open a separate issue for the MySQL / OceanBase DDL gap (emit `NOT NULL` for primary-key columns, with a unit test in `MysqlCreateTableSqlBuilderTest`). Happy to help review it, and a follow-up PR from you would be very welcome.
**Where this leaves the PR**
- This red job is caused by the PR's own new E2E, so it is not an unrelated flake and a rerun will not clear it. It needs the fixture change above (or option 1) before the checks can go green.
- My earlier approval was about the code-level review of the change; it does not cover this failing E2E, so please treat the jdbc-connectors-it-part-1 job as a merge gate that still needs to pass on the new head.
- Once you push the fixture change, please share the new check link here if anything else fails and I will help narrow it down.
Thanks again for the careful diagnosis, this is a nice piece of work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5642818292
@luozihen thanks for the detailed per-item status — this looks like the right direction.
- **F1** — Good. Keeping `ReadonlyConfig#toConfig()` unchanged and scoping the fix to `MultiTableFailureHelper.mergeOptions()` removes the cross-cutting compatibility risk. One ask: since `mergeOptions()` now merges the option maps directly, could you confirm (ideally with a test) that a dotted key present in both the shared options and a per-table override resolves the same way as before the PR?
- **F2 / F5** — The `LinkedHashMap<Pattern, List<String>>` in `toCompiledPatternMap` plus `testResolveMultiTablePrimaryKeysFirstMatchWins` addresses the core concern. The remaining question is upstream of that method: the map that `toCompiledPatternMap` receives must itself preserve HOCON declaration order, otherwise the `LinkedHashMap` only preserves an already-scrambled order. Could you point me to where that is guaranteed, or add a test that declares two overlapping patterns in non-alphabetical order in a real config string and asserts the first-declared one wins?
- **F3** — Looks good. Rejecting blank names and names containing a comma in `validatePrimaryKeyColumns` is the minimum needed to keep the join safe, and `testFactoryContextWithMultiTableConfigInvalidColumnFails` covers it.
- **F4** — Your comment appears to be cut off mid-sentence at the `TablePlaceholderProcessor` part, so I can't see the conclusion. Could you re-post the rest? Specifically: which expansion runs first, and what happens if a user writes `${primary_key}` in `multi_table_config.primary_keys` for a table whose catalog has no primary key — does the connector fall back, fail fast, or leave the literal placeholder in `PRIMARY_KEYS`?
Items not covered in your update:
- **F6** — Is there explicit validation of user-supplied regex patterns at factory time, mapped to the JDBC-12 error code, with a test for an invalid pattern?
- **F7** — With F1 narrowed, the `toConfig()` regression test is less critical, but an IT/E2E exercising the new multi-table option end-to-end through the engine (not only the factory unit tests) would still be valuable. Is one included?
- **F8** — Your comment consistently uses `multi_table_config`; please confirm the option key in `JdbcSinkOptions` and the docs is `multi_table_config` throughout and the mixed `multi-table_config` spelling is gone.
Once F4 and F6–F8 are confirmed, I'm happy to take another pass.
<!-- streview-comment:977 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5658593905
@DanielLeens Thanks for independently verifying F1 and for pinning down the merge semantics precisely.
Agreed — the change is a narrowing from HOCON-style recursive `withFallback` to a shallow top-level
merge, and your call-site analysis matches mine: no current caller can hit the diverging case.
I've added a Javadoc note on `mergeOptions()` stating that the merge is shallow/top-level and that,
on a key collision, `primary` replaces `fallback`'s value wholesale rather than recursively merging
nested objects. Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5852837915
@SEZ9 thanks for the quick follow-up. Two things to close out.
**On the "cut off" comment** — I re-fetched my previous comment directly through the GitHub API (not the rendered web page) and it is complete on the server side; nothing was truncated in storage. My best guess is the web UI's long-comment fold ("Show more") only rendered the visible portion for you. For the record, here is the tail that answers (a) and (b) directly:
> ...declared in that non-alphabetical order — the catch-all `^table.*$` is declared last even though it would sort first — builds a `ReadonlyConfig` via `ReadonlyConfig.fromConfig(...)` from the parsed `Config`, and calls the production `factory.resolveMultiTablePrimaryKeys(...)` directly, not a hand-built map. It asserts `table1` resolves to `["c_int", "c_integer"]` and `table2` to `["c_mediumint"]` — the first-declared pattern wins for both, not the catch-all.
I went back to the source again to answer your two specific questions precisely:
**(a) Does the exact-match pattern win over the catch-all declared last?** Yes. `JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins` (`JdbcSinkFactoryTest.java:347-384`) declares the HOCON map as `^table2$`, then `^table1$`, then the catch-all `^table.*$` last. If iteration order were not preserved (e.g. if it fell back to alphabetical or hash order), the catch-all would win for both tables since it sorts first and matches both. The test asserts `table1 -> [c_int, c_integer]` and `table2 -> [c_mediumint]`, i.e. the specific pattern wins in both cases, not `[c_smallint]` from the catch-all. So non-alphabetical declaration order is honored end to end from a parsed `Config` through `ReadonlyConfig` into `resolveMultiTablePrimaryKeys`.
**(b) Does the order guarantee hold through the shade round trip, not just in the factory unit test?** Yes, and it's actually verified at three independent layers, not only the one you quoted:
1. `JdbcSinkFactoryTest.java:347-384` — proves order survives `HOCON string -> ConfigFactory.parseString -> ReadonlyConfig.fromConfig -> resolveMultiTablePrimaryKeys`, as above.
2. `ConfigShadeTest#testDecryptPreservesSpecialCharacterKeys` (`ConfigShadeTest.java:439-468`) — exercises `ConfigShadeUtils.decryptConfig(...)` directly (this is the method that wraps `processConfig`, the JSON-round-trip shade step) on a config containing two `multi_table_config.primary_keys` regex keys, and asserts both regex keys and their list values survive the round trip unchanged.
3. `JdbcMysqlMultipleTablesIT#testMysqlJdbcMultipleTableE2e` runs the real engine job-submission pipeline end to end, using the same non-alphabetical, catch-all-last pattern set (`^mtc_table2$`, `^mtc_table1$`, `^mtc_.*$` — see the test resource config), and asserts the primary keys read back from the database catalog match the specific patterns (`mtc_table1 -> [c_int, c_integer]`, `mtc_table2 -> [c_mediumint]`), not the catch-all. Since `ConfigShadeUtils.processConfig` runs unconditionally at job submission (`ConfigShadeUtils.java:143` -> `:152`), this E2E test necessarily exercises the shade round trip too, so it closes the gap between "unit-tested in isolation" and "actually holds in the full submission path."
So F2/F5 is covered end to end across the factory unit test, the shade unit test, and the E2E test — I'd consider it closed with solid evidence, not just a single narrow assertion.
**On the F1 regression-test ask** — good catch, but it's already there and I should have pointed to it explicitly instead of only citing the positive case. `ConfigShadeTest#testDecryptPreservesOrdinaryConfigShapes` (`ConfigShadeTest.java:470-496`) is exactly the negative-case regression test you're asking for: it builds a config with an unquoted `split.size` key (which HOCON expands into a nested object) and a quoted literal `"dfs.replication"` key inside a nested map, runs it through `ConfigShadeUtils.decryptConfig(...)`, and asserts the `env`/`source`/`sink` blocks come out byte-for-byte equal (via `root().unwrapped()`) to the input. Paired with the positive regex-key case in `testDecryptPreservesSpecialCharacterKeys` (`ConfigShadeTest.java:439-468`), both the "must not change" and "must now work" halves of the JSON round trip are covered. I don't think a new test is needed here; the existing pair already asserts exactly what you described.
With that, I don't have any open items on F1/F2/F5 either. Still Ready to merge from my side, pending CI and a write-capable maintainer for the merge action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4320486484
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
I've made the changes according to your guidance, and CI is green again. Please feel free to review it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6406668724
Hi @hutiefang76 — following up on the contract question from 23 Oct. I have not heard back for a few days, so rather than leave this open I will proceed in the direction @SEZ9 and @DanielLeens converged on, and I wanted to tell you explicitly first.
1. The disputed row is settled by the parser itself. ConfigParseOptions.PATH_TOKEN_SEPARATOR is "->" in this repo (seatunnel-config-shade/.../ConfigParseOptions.java:23), not ".". That is why parseMap keeps job.mode / shade.identifier as literal keys (and env.hasPath("shade.identifier") is true), while a->b is the form that expands, and why ^t_.*$ still fails with BadPath. I will pin this with the REST-shaped regression and a direct toConfig() case, and align the wording of the existing test with it.
2. I will keep the REST fix in this PR, but narrow it. @nzw921rx asked for the REST JSON path to work here, and @SEZ9 / @DanielLeens both prefer the narrow contract, so I will replace the all-literal rebuild with quote-only-invalid: a key is quoted only when it cannot be parsed as a path, everything that parses today keeps its current behavior (a->b still expands, dotted keys and quoting behave as before), and regex keys become representable. This is the narrow option we discussed, not the broad literal rebuild.
3. Ownership. ConfigBuilder.of(Map), backfillUserVariables and ConfigShadeUtils.processConfig will share one narrow helper, so the contract lives in one place. For #12443, I suggest it rebases onto this and keeps the parts that go beyond this PR (ReadonlyConfig.toConfig(), the config-adapter path and both variable-map conversions), dropping the overlapping pieces. If you would rather keep #12443 as the only owner of the whole conversion, tell me within the next two days and I will drop 0c43f10dd instead and rebase once #12443 lands.
4. Tests in this PR. The REST-shaped case (map -> RestUtil.buildConfig -> ConfigShadeUtils.decryptConfig) with shade.identifier, job.mode, a.b, a->b and a regex key, asserting which keys expand and which stay literal; plus a direct toConfig() case in ReadableConfigTest so both entry points are pinned.
Thanks — I would like to settle this and move the PR forward; if you disagree with either the contract or the ownership, a short reply is enough and I will adapt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub) via commits" <co...@seatunnel.apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-7033449067
@nzw921rx @SEZ9 @DanielLeens — following the after-3.1.0 plan, this branch is now synced with the 3.1.0 dev branch and #12443's config-key contract, and the Build is green.
Head: 2c899fc6c (merge of upstream/dev f1ab36f04, which contains #12443)
Build: https://github.com/luozihen/seatunnel/actions/runs/44469734647 — success, including the two all-connectors partitions that previously timed out (they passed on the final attempt).
What changed:
- ConfigBuilder.of(Map), ConfigBuilder.backfillUserVariables and ConfigShadeUtils.processConfig now come from #12443 (ConfigMapPathUtils.quoteInvalidPathKeys). `git diff upstream/dev` shows no changes in those files, so this PR no longer carries a competing conversion.
- The MultiTableFailureHelper.mergeOptions workaround and its test were removed, since ReadonlyConfig.toConfig() is fixed upstream.
- The JDBC feature, the fail-fast/empty-key validation, the docs and the regression tests remain.
Local verification: ConfigBuilderTest 21, ConfigShadeTest 20, ReadableConfigTest 18, JdbcSinkFactoryTest 47, JdbcTableOptionsConditionExtensionTest 15 — all pass; spotless check passes.
One CI bookkeeping note: the PR currently shows no attached Build check because the `PR signal` run for this head is still `action_required`, so please use the fork run link above. Ready for re-review whenever you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5578424721
@luozihen Great news, thanks for rerunning — glad the Doris flake cleared and the run is green now. No need to apologise, you're doing everything right for a first PR.
Next steps from here:
1. I'll do the fresh confirmation pass I promised on `1730aeef`, going back through the points from my earlier review. To make that quick and avoid me guessing, could you post a short checklist in this thread of which of those items your last push (the HOCON fix + rebase) addressed, and which are still open or you'd like to push back on? Specifically:
- **F1** — the `ReadonlyConfig.toConfig()` JSON round-trip and its effect on dotted-key path semantics for all callers.
- **F2 / F5** — the "first pattern in declaration order wins" guarantee vs. the `Map<String, Object>` option type, and keeping the doc statement in `docs/en/connectors/sink/Jdbc.md` consistent with whatever ordering the code actually guarantees.
- **F3** — identifier validation for the resolved key columns before they're comma-joined into `PRIMARY_KEYS` and reach generated SQL.
- **F4** — how the connector-level `${primary_key}`/`${unique_key}` expansion interacts with the engine-level `TablePlaceholder` replacement.
- **F6** — explicit compile-time validation of user-supplied regex patterns mapped to the new JDBC-12 error code.
- **F7** — a regression test for the previous `toConfig()` dotted-key expansion and an engine-level IT/E2E for the new option.
- **F8** — the `multi-table_config` key naming (hyphen + underscore mix).
2. Once I've confirmed those, if anything is still open I'll leave targeted comments; if it's all covered I'll approve and we'll get a second maintainer look before merge, which is standard here.
You don't need to take any further action on CI. Just the checklist above and I'll take it from there.
<!-- streview-comment:896 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5747177276
Hi @DanielLeens — I need your advice on how to handle a CI failure before I make any further changes, because I would rather not widen this PR unless you think that is the right call.
What fails. jdbc-connectors-it-part-1 fails on the engine-level E2E I added — JdbcMysqlMultipleTablesIT#testMysqlJdbcMultipleTableE2e, the assertion at line 210. That phase drops sink.table1 / sink.table2 so the sink creates them itself, which is what makes the resolved multi_table_config.primary_keys observable in the created DDL.
Why it fails — this looks like a pre-existing gap, not the mapping logic. The key columns configured for those tables (c_int, c_integer, c_mediumint) are nullable in the source metadata, and MysqlCreateTableSqlBuilder#buildColumnIdentifySql emits an explicit NULL for every nullable column (MysqlCreateTableSqlBuilder.java:204-209). The generated DDL therefore ends up as:
`c_int` int(11) NULL COMMENT '',
`c_integer` int(11) NULL COMMENT '',
PRIMARY KEY (`c_int`, `c_integer`)
MySQL rejects that: sql/sql_table.cc raises ER_PRIMARY_CANT_HAVE_NULL (1171, "All parts of a PRIMARY KEY must be NOT NULL") when a primary key column is explicitly declared NULL — if the keyword is omitted, MySQL converts the column to NOT NULL implicitly. AbstractJdbcCatalog#createTableInternal then wraps it as Failed creating table sink.table1 (Caused by: java.sql.SQLSyntaxErrorException: All parts of a PRIMARY KEY must be NOT NULL ... in the log) and the job fails.
So I do not think this is a defect in the new option: existing auto-create cases never hit it because their key columns are NOT NULL (CDC primary keys, and JdbcMySqlCreateTableIT uses id int NOT NULL), and the Postgres builder omits the nullability keyword entirely, which is why the same scenario does not fail there.
What I would like your guidance on. I see two ways to make CI green and I do not want to pick the wrong one:
1. Fix it here: make MysqlCreateTableSqlBuilder emit NOT NULL for columns that are part of the primary key, plus a unit test in MysqlCreateTableSqlBuilderTest. The E2E then stays exactly as it is and doubles as the end-to-end regression. The change only turns "currently fails" into "works" — it does not alter DDL for configurations that already work.
2. Keep this PR scoped to multi_table_config: adjust the E2E so its key columns are NOT NULL (or use dedicated tables for that phase, so the shared fixture stays untouched) and track the MySQL DDL gap separately. The test would then not depend on that pre-existing issue.
My own preference is (2) — keeping this PR focused on the new option and handling the builder fix separately — but I am happy to do (1) here if you think that is cleaner, or to change how the E2E observes the mapping if you would rather it did not depend on auto-created tables at all.
Could you let me know which you would prefer? And if we go with (2): is there an existing issue for the MySQL DDL gap I should reference, or should I open one?
For completeness: unit-test passes on all four lanes for this head (c15f9fdc5), so this is specific to the E2E auto-create path. CI: [run 35303656875 / jdbc-connectors-it-part-1](https://github.com/luozihen/seatunnel/actions/runs/35303656875/job/105471901217). This PR: [#12015](https://github.com/apache/seatunnel/pull/12015).
Thanks — apologies for yet another question on this PR; I just want to avoid growing the diff in a direction you would not choose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5696080027
@luozihen Good diagnosis on your part — you've already found the right error, and it's not something in your control to fix by changing test logic. Confirmed it directly from the fork's run logs, so here's the full picture rather than a "just retry" answer.
**Root cause, confirmed from logs (not a guess):** Docker Hub's `minio/minio` repository was removed entirely back in mid-September — it's a 404 `pull access denied ... repository does not exist`, not a rate limit or transient network blip, and it will never succeed no matter how many times the job is rerun. I pulled the actual failed-job logs from your fork's run (`luozihen/seatunnel` run `34802808218`, run-attempt 12) and confirmed the same signature on all 8 failing legs:
```
Status 404: {"message":"pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}
```
— hit by `PaimonWithS3IT.startUp` (`paimon-connector-it`), `DatabendIT` / `DatabendCDCSinkIT` (`all-connectors-it-2`/`-6`/`-7`), and the same pattern on the other failing legs — every one of them a test that spins up a MinIO Testcontainer, unrelated to the actual test assertions.
**This is not caused by your diff.** Your PR only touches `seatunnel-api`, `connector-jdbc`, `seatunnel-core-starter`, and docs — none of the failing test classes (Paimon/Databend/S3 E2E) are anywhere in your changed-files list, so there's nothing in your JDBC primary-key-mapping change that could produce this.
**It's already fixed on `dev`, your branch just doesn't have it yet.** This exact issue hit a large batch of PRs a few days ago and was fixed in apache/seatunnel#12287 (merged 2026-09-13), which swaps the image reference to `quay.io/minio/minio` (Docker Hub's `minio/minio` repo is gone for good, not coming back). Your branch head (`2c5982be8`) is currently 174 commits behind `dev` — it predates that fix, which is exactly why every rerun keeps failing the same way.
**What to do:** no code change needed on your side, and no need to touch your actual feature logic — just pick up the fix from `dev`:
```bash
git fetch origin dev
git rebase origin/dev
# resolve any real conflicts (unlikely — the fix only touches unrelated E2E test files)
git push --force-with-lease
```
That should bring in the `quay.io/minio/minio` swap and clear the Paimon/Databend/S3 failures. Ping me once CI's green again on the new head and I'll take a fresh look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5755023969
@SEZ9 Thanks — to stop the truncation we keep hitting, I have consolidated every open item into the
"Review status (F1-F8)" section of the PR description, with the file and test name for each.
Could you use that section as the source of truth for the next pass? One-line index below, in case
it helps:
- F1 — `ReadonlyConfig#toConfig()` is unchanged (`ConfigFactory.parseMap(confData)`,
`ReadonlyConfig.java:76`) and `ReadonlyConfig.java` is not in the PR diff at all; the parsing fix
is scoped to `ConfigShadeUtils.processConfig` and `MultiTableFailureHelper#mergeOptions`.
- F2 / F5 — explicit `LinkedHashMap<Pattern, List<String>>` in `toCompiledPatternMap`, so the order
is guaranteed by construction, plus
`JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins`, which uses a real
HOCON string with non-alphabetical overlapping patterns; both docs state the same rule.
- F3 — `validatePrimaryKeyColumns` rejects blank names and commas with `JDBC-12`, and the resolved
names are quoted through the dialect when SQL is generated, so those two are the only inputs that
cannot be represented safely.
- F4 — the order is documented in
`docs/{en,zh}/introduction/configuration/sink-options-placeholders.md` and pinned by
`JdbcSinkFactoryTest#testPlaceholderOrderBetweenEngineAndConnectorPasses`.
- F6 — patterns are compiled once up front (`compilePattern` + bounded cache) and an invalid
pattern raises `JDBC-12` with the offending pattern in the message; three tests cover it.
- F7 — the `toConfig()` half is not open: the regression test already exists in this PR's diff
(`ReadableConfigTest#testToConfigPreservesDottedKeyExpansion`, `seatunnel-api` line 383). The E2E
half is now updated exactly as you asked in `2656c4b6a`: two dedicated source tables
(`source.mtc_table1` / `source.mtc_table2`) whose key columns are `NOT NULL`, the `.conf` pointed
at them, and the shared `source.table1` / `source.table2` left untouched.
- F8 — confirmed: `multi_table_config` is used consistently in `JdbcSinkOptions`, both docs, the E2E
config and all tests.
The MySQL/OceanBase DDL gap is not fixed here, as you suggested. I will open a separate issue for it
(emit `NOT NULL` for key columns plus a unit test in `MysqlCreateTableSqlBuilderTest`) and link it
here once it is created, then follow up with a PR.
`jdbc-connectors-it-part-1` on `2656c4b6a` is the check to watch; I will paste the link here once it
finishes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5770549590
@SEZ9 Thanks — I've added the three clarifications to the "Review status (F1-F8)" section of the PR description: F1 now states that neither parsing change is specific to this option and explains why it is behaviour-neutral for every other option (plus the mergeOptions callers), F2/F5 now spells out the exact chain the ordering test uses (HOCON string → ReadonlyConfig → option → toCompiledPatternMap), and F3 now answers the quote-character question deliberately (the dialect quoting wraps without escaping, so such names cannot come from catalog expansion; a dialect-specific character rule would be a follow-up). F4/F6/F7/F8 name the tests and the final key name as you asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5488719462
@luozihen Thanks for flagging this, and no worries about the noise.
Before answering the revert-vs-rebase question I dug into the actual CI failure, because fixing the merge commit alone will **not** fix CI here.
**The real root cause is not the upstream `dev` merge — it's a HOCON syntax bug in your own new test**, `ConfigShadeTest#testDecryptPreservesSpecialCharacterKeys` (`seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java:407-416` at your current head). I compared this method between your pre-merge commit (`88c2358193b9`) and the merge commit (`2acb55a048d0`) and it is byte-for-byte identical in both — so the `apache:dev` merge did not introduce this failure, it was already broken before you merged.
The failing job is `unit-test` (all four JDK/OS matrix lanes), failing in module `seatunnel-core-starter`:
```
[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0 <<< FAILURE! - in org.apache.seatunnel.core.starter.utils.ConfigShadeTest
[ERROR] testDecryptPreservesSpecialCharacterKeys Time elapsed: 0.004 s <<< ERROR!
org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Parse: String: 2: Expecting close brace } or a comma, got '='
at ConfigShadeTest.testDecryptPreservesSpecialCharacterKeys(ConfigShadeTest.java:442)
```
The raw HOCON literal the test builds its input from is:
```java
"source { FakeSource { plugin_output = \"fake\" } }\n"
+ "sink { Jdbc { url = \"jdbc:mysql://localhost:3306/db\" "
+ "username = \"u\" password = \"p\" "
+ "multi-table_config { primary_keys { "
+ "\"^t_nova_.*$\" = [\"${primary_key}\", \"DATA_SOURCE\"] "
+ "\"^t_tyuen_txn_.*$\" = [\"id_txn_ctrl\", \"DATA_SOURCE\"] "
+ "} } } }"
```
In HOCON, fields inside an object must be separated by a comma or a newline. `url = "..." username = "u"` on one physical line with only a space between them is not valid — the parser tries to concatenate the quoted `url` value with the following unquoted token `username`, then errors out on the `=` that comes right after it (hence "Expecting close brace } or a comma, got '='"). The same missing-separator problem repeats for `username`/`password`, `password`/`multi-table_config`, and the two `primary_keys` entries (`"^t_nova_.*$" = [...]` next to `"^t_tyuen_txn_.*$" = [...]`).
Fix: add commas (or real newlines) between every field, e.g.:
```java
Config input =
ConfigFactory.parseString(
"source { FakeSource { plugin_output = \"fake\" } }\n"
+ "sink { Jdbc { url = \"jdbc:mysql://localhost:3306/db\", "
+ "username = \"u\", password = \"p\", "
+ "multi-table_config { primary_keys { "
+ "\"^t_nova_.*$\" = [\"${primary_key}\", \"DATA_SOURCE\"], "
+ "\"^t_tyuen_txn_.*$\" = [\"id_txn_ctrl\", \"DATA_SOURCE\"] "
+ "} } } }");
```
(or switch to a Java text block / genuinely multi-line literal so real newlines do the separating for you — that tends to be less error-prone than tracking commas across a long chain of concatenated strings).
On your actual question, once that test is fixed:
- **Recommendation: rebase (your option 2), not revert.** This is your own PR source branch, not a shared/protected branch, so an interactive rebase to drop the accidental merge commit followed by a force-push is completely normal and safe here — that's exactly what rebasing your own feature branch is for. `git revert -m 1 <merge-sha>` also "works," but reverting a merge commit has a well-known follow-up trap: if you ever merge `apache:dev` into this branch again later, git can silently treat the commits from the first merge as already-integrated-then-reverted, and you'd have to revert your own revert first to bring them back. Rebasing avoids that trap and gives reviewers a clean, linear diff.
- Practically, it won't affect the permanent project history either way: Apache SeaTunnel merges PRs with squash-merge, so everything on this branch collapses into a single commit on `dev` regardless of whether the branch itself contains a merge commit, a revert commit, or a clean rebase.
So: rebase onto the latest `apache:dev`, fix the missing commas in `testDecryptPreservesSpecialCharacterKeys`, force-push to your own branch, and CI should go green. Let me know if anything else comes up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6306844335
Thanks @luozihen for the follow-up on `82fc5e61a`.
The thread already has pointers for the first few of my earlier points, so I won't re-ask those:
- **F1 / F7 (`ReadonlyConfig.toConfig()` round-trip):** `ReadonlyConfig.java` unchanged, with `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` pinning the behaviour and the regex-key fix moved into `ConfigBuilder.rebuildWithLiteralKeys` / `MultiTableFailureHelper.mergeOptions`.
- **F2 / F5 (declaration order):** `LinkedHashMap` in `toCompiledPatternMap`, covered by `JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins`, with the en/zh docs stating the same rule.
- **F3 (identifier validation):** `validatePrimaryKeyColumns` running before the comma-join, covered by `testFactoryContextWithMultiTableConfigInvalidColumnFails`.
- **F4 (placeholder interaction):** documented in `Jdbc.md` and `sink-options-placeholders.md`.
I'll check those against the diff on this head. A few items are still open from my side:
1. **F6 (regex validation):** Please confirm that an invalid user-supplied pattern fails with `JDBC-12` when the pattern is compiled, and point me to the test covering it.
2. **F7 (engine-level coverage):** Could you link the E2E case/config that exercises the new option through the engine rather than only the factory?
3. **F8 (option key naming):** Is the user-facing key still `multi-table_config`? If so, I'd still like it aligned with the JDBC sink's snake_case convention before merge; if it has already been renamed, just tell me the final key.
Once those are clear, I'm happy to move this forward.
<!-- streview-comment:1912 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6431953601
@luozihen — apologies for the slow reply on the contract question; answering it directly now.
1. Agreed on the parser point. Given `PATH_TOKEN_SEPARATOR` is `->` in this repo, the row we disagreed on resolves the way you describe: `job.mode` / `shade.identifier` stay literal through `parseMap`, `a->b` is the form that expands, and `^t_.*$` is the one that actually breaks with `BadPath`. That settles PR12015-F1 in favour of the narrow fix rather than the all-literal rebuild, so I withdraw my objection to keeping the REST fix in this PR.
2. Quote-only-invalid is the contract I'm comfortable with: quote a key only when it cannot be parsed as a path, and leave everything that parses today untouched. Please make sure the implementation really is a no-op for every key that parses today (dotted keys, already-quoted keys, `a->b`), so no existing caller of `toConfig()` sees a change.
3. On tests (PR12015-F7), what I need to see before I can resolve this:
- the REST-shaped regression (`map -> RestUtil.buildConfig -> ConfigShadeUtils.decryptConfig`) plus a direct `toConfig()` case, asserting `shade.identifier`, `job.mode`, `a.b`, `a->b` and a regex key together in one test so the final behavior is pinned in one place;
- the existing test's wording aligned with the `->` separator, as you proposed.
4. Sequencing: since #12443 (head `509fddab`) carries the config-key conversion, please land that first, then sync `dev` and drop `0c43f10dd` from this branch so the two PRs don't ship competing conversions. The head here is still `d9bb1574`, so my review stands until the rebased head is pushed; once it is, I'll re-review the REST point and the test together.
@nzw921rx noted — fine to take this after 3.1.0; nothing here needs to rush the release.
Nothing further is needed from you until #12443 merges. Ping me when the rebased head is up.
<!-- streview-comment:2054 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-7050632252
@luozihen thanks for the sync and the clear summary.
The latest review on this thread is an approval that lists no blocking source items and says the existing review suggestions are unchanged. So everything below is a non-blocking follow-up from the earlier review, not a new requirement.
**CI:** you report fork run 44469734647 as successful for head 2c899fc6c. Because the `PR signal` run is still `action_required`, that result is not yet an attached Build check on this PR. The workflow approval is a repository-side step.
**Not confirmed from the comments alone**
- **`ReadonlyConfig.toConfig()` dotted-key semantics:** your summary says `git diff upstream/dev` shows no changes in the `ConfigBuilder` and `ConfigShadeUtils` code, and that the `MultiTableFailureHelper.mergeOptions` workaround was removed. It does not say whether this PR still changes `ReadonlyConfig` itself relative to dev. Could you confirm that, so the earlier concern about path semantics for other callers can be closed?
**Earlier suggestions your comment doesn't cover**
1. **Pattern order:** the option is a `Map<String, Object>` and HOCON object keys don't guarantee declaration order. Is "the first pattern in declaration order wins" now guaranteed in code, or were the docs reworded?
2. **Identifier validation:** are the resolved key columns validated before they are comma-joined into `PRIMARY_KEYS` and reach the generated SQL?
3. **Placeholder interaction:** how does the connector-level `${primary_key}` / `${unique_key}` expansion interact with the engine-level `TablePlaceholder` replacement, and is that tested or documented?
4. **Regex validation:** does the fail-fast/empty-key validation also compile user-supplied patterns up front and map failures to the new JDBC error code?
5. **Tests:** does `ReadableConfigTest` include a regression case for the old dotted-key expansion, and is there an engine-level IT/E2E for the new JDBC option?
6. **Option name:** is `multi-table_config` staying as is, or will it be renamed to match the JDBC sink's snake_case convention?
A short "done / not done / intentionally kept" per item is enough, with a pointer to the relevant test or doc section where one exists.
<!-- streview-comment:2479 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5488653844
Hi @luozihen — thanks for flagging this, but good news: you don't actually need to revert or rebase to fix CI here. I traced the failure before answering, and the merge commit isn't the cause.
The real failure is a genuine, 100% reproducible bug in the new unit test added by this PR itself:
`ConfigShadeTest.testDecryptPreservesSpecialCharacterKeys` (`seatunnel-core/seatunnel-core-starter/src/test/java/org/apache/seatunnel/core/starter/utils/ConfigShadeTest.java:407-416`) fails in the `unit-test` job on all 4 JDK/OS legs (8/11 x ubuntu/windows) with:
```
org.apache.seatunnel.shade.com.typesafe.config.ConfigException$Parse: String: 2: Expecting close brace } or a comma, got '=' ...
```
Root cause is a HOCON syntax bug in the literal string passed to `ConfigFactory.parseString(...)`:
```java
"sink { Jdbc { url = \"jdbc:mysql://localhost:3306/db\" "
+ "username = \"u\" password = \"p\" "
+ "multi-table_config { ... } } }"
```
In HOCON, sibling key-value pairs on the same line need a comma (or a newline) between them. Since `url = "..."`, `username = "u"`, and `password = "p"` sit on one logical line separated only by whitespace, the parser treats the quoted URL string and the following bareword `username` as one concatenated value, then chokes on the unexpected `=` right after it — which matches the "got '='" message exactly. This is a static parse error, not timing/environment dependent, so it fails on every run regardless of the merge commit.
Fix: add commas (or put each field on its own line) between `url`/`username`/`password`/`multi-table_config` in that HOCON literal, e.g.:
```java
"sink { Jdbc { url = \"jdbc:mysql://localhost:3306/db\", "
+ "username = \"u\", password = \"p\", "
+ "multi-table_config { ... } } }"
```
Separately, the other failure in this run (`rocketmq-connector-it`) is an unrelated Testcontainers/Docker networking flake inside the RocketMQ broker container (`RemotingSendRequestException: send request to ... failed`) — unrelated to your diff, safe to ignore/rerun.
On the git-history question: this repo squash-merges PRs at merge time, so the intermediate commit history (merge commit included) won't carry into the final merged commit either way — you don't need to revert or force-push-rebase purely for CI's sake. If you'd still like a tidier branch, `git revert -m 1` (your option 1) is the safer choice since the merge commit is already pushed publicly. Push the test fix above on top of that, and CI should go green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5490145930
@DanielLeens
Thank you very much for your patience and thoroughness in diagnosing the exact cause of the CI failure – I truly appreciate it. This was indeed my oversight: I verified the changes using my usual testing approach but did not actually run this specific unit test locally, which led to the issue. I have now revised the HOCON block in the configuration literal and verified that the test passes locally. I have pushed the updated code and am now waiting for the CI to finish. Again, I apologise for the inconvenience and am very grateful for your detailed guidance.
Separately, I have rebased my branch as suggested. Going forward, I will avoid merging from the upstream repository into my feature branch, as that can introduce unnecessary noise and distract from the actual changes in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5534300733
@DanielLeens Quick follow-up: the latest CI run has finished, and it failed on a different set of errors — all
in the Doris connector integration tests:
```text
DorisCDCSinkIT » ... org.testcontainers.containers.ContainerLaunchException ...
DorisErrorIT.testDoris:91 » ContainerLaunch Container startup failed
DorisIT » ... org.testcontainers.containers.ContainerLaunchException ...
```
These are ContainerLaunchException / Container startup failed from Testcontainers, which looks
like the Docker container couldn't be started in the CI environment — similar in nature to the
earlier Docker Hub pull timeout. They're in the Doris connector ITs and don't appear related to this
PR, whose changes are in the JDBC connector and config modules.
I'm still not entirely sure how to interpret these, so if you could let me know whether this is
likely a transient CI/Docker environment issue or something I should address on my side, I'd really
appreciate it.
Thanks again for your patience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5658245295
@SEZ9
Re-posting the parts that were cut off:
**F4** — The engine-level `TablePlaceholderProcessor.replaceTablePlaceholder` runs first (during
`TableSinkFactoryContext.replacePlaceholderAndCreate`, before `createSink`), but it only rewrites
top-level String and single-element-String-List values, so it leaves the nested `multi_table_config`
map untouched. The connector-level `expandPrimaryKeyPlaceholder` then expands `${primary_key}` /
`${unique_key}` inside `multi_table_config.primary_keys`. If a matched table uses `${primary_key}`
but the upstream catalog has no primary key, the connector fails fast with `JdbcConnectorException`
(`JDBC-12`) — it does not fall back and does not leave the literal placeholder in `PRIMARY_KEYS`.
**F6** — Yes. `toCompiledPatternMap` compiles every declared pattern up front via `compilePattern`;
an invalid regex raises `JDBC-12` immediately. Covered by `testResolveMultiTablePrimaryKeysInvalidRegexFails`
and `testCompilePatternInvalidFails`.
**F7** — The `toConfig()` dotted-key regression test is present
(`ReadableConfigTest#testToConfigPreservesDottedKeyExpansion`). The engine-level IT/E2E for the new
option is deferred and noted as a follow-up in the PR description.
**F8** — Confirmed: the option key is `multi_table_config` (snake_case) across `JdbcSinkOptions`,
error strings, unit tests, and EN/ZH docs; the mixed `multi-table_config` spelling is gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673729055
@SEZ9 F6 — Both tests are part of this PR:
`JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysInvalidRegexFails` (invalid pattern through
`resolveMultiTablePrimaryKeys`) and `JdbcSinkFactoryTest#testCompilePatternInvalidFails` (invalid
pattern through `compilePattern`). Both assert `JdbcConnectorException`, i.e. the JDBC-12 path;
compilation happens up front in `toCompiledPatternMap` -> `compilePattern`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5862393420
@luozihen Thanks for the detailed report and for the retries. The failure in `all-connectors-it-2 (8, ubuntu-latest)` / `all-connectors-it-2 (11, ubuntu-latest)` — not being able to pull the `quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z` image — looks like an image-pull problem in the test environment rather than something caused by this PR's changes. Since eight retries gave the same result, it doesn't look like a simple transient blip, so there's no need to keep retrying for now. I'll take a look at that side and re-trigger those jobs once it's sorted; we won't block the PR on a purely environmental error.
In the meantime, the review items that still need attention on the PR itself are:
- **F1 / F7 (ReadonlyConfig.toConfig())** — the JSON round-trip changes how dotted keys are expanded for every caller of `toConfig()`, not only the JDBC sink. Please either scope the change so existing callers keep the old behavior, or add a regression test in `ReadableConfigTest` that pins the previous dotted-key expansion and shows why the new behavior is safe.
- **F2 / F5 (declaration-order matching)** — the option is a `Map<String, Object>`, so "first pattern in declaration order wins" isn't guaranteed. Either switch to an ordered structure (e.g. a list of pattern entries) or update `Jdbc.md` to describe the actual matching rule.
- **F3 (PRIMARY_KEYS into SQL)** — resolved key column names are comma-joined and end up in generated SQL; please validate them as identifiers (or check them against the table schema) before they reach the SQL builder.
- **F4 (`${primary_key}` / `${unique_key}` vs. engine TablePlaceholder)** — please clarify in code/docs which layer expands these placeholders and what happens when both apply.
- **F6 (regex validation)** — compile user-supplied patterns up front and map invalid patterns to the new `JDBC-12` error code with a clear message.
- **F7 (E2E)** — add at least one engine-level IT/E2E case exercising the new option end-to-end.
- **F8 (option name)** — `multi-table_config` mixes hyphen and underscore; please rename to a consistent snake_case key.
Once those are pushed I'll re-run CI and do another pass.
<!-- streview-comment:1371 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6338270536
@nzw921rx @DanielLeens @SEZ9
CI is red. After checking the logs, I confirmed that it failed in two places:
- Issue 1:There is a deadlink in the documentation:
https://raw.githubusercontent.com/intersystems-community/iris-driver-distribution/main/JDBC/JDK18/intersystems-jdbc-3.8.4.jar
- Issue 2:unit-test(ubuntu/windows,11/17) reports an error:
ERROR org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine - Open udf failed failed
java.lang.Exception: open failed
Issue 1 should be caused by merging code, and issue 2 should be intermittent.
For issue 2, I can recompile. For issue 1, how can I handle it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6329655732
Thanks @nzw921rx for re-checking the current head (`82fc5e61ad`) so carefully, and thanks @luozihen for the patience through these rounds. I re-read both points against the code.
**1. Empty resolved key list: you are right, and I should have caught this. Sorry for missing it in my earlier approvals.**
`JdbcMultiTableConfigConditionExtension` only checks that the `primary_keys` map itself is non-empty. For `"^table1$" = []`, `toPrimaryKeyList()` returns an empty list, `expandPrimaryKeyPlaceholder()` returns it unchanged, and `applyPrimaryKeys()` calls `validatePrimaryKeyColumns()`, whose loop is a no-op for an empty list. The sink then builds a `PrimaryKey` with no columns, and `JdbcSink.createWriter()` reaches `getColumnNames().get(0)` (`JdbcSink.java:145-146`), which throws `IndexOutOfBoundsException`. The `null` element case is also real: `String.valueOf(element)` turns it into the column name `"null"`, which `validatePrimaryKeyColumns()` does not reject.
This is an accepted config that fails with an obscure runtime error, so I agree it should be fixed before merge. @luozihen, a small change should be enough:
- Reject an empty resolved list for a matched table with `JDBC-12` (for example at the top of `validatePrimaryKeyColumns()`, or earlier in `toPrimaryKeyList()`), naming the pattern/table in the message.
- Reject non-string or `null` list elements in `toPrimaryKeyList()` with `JDBC-12` instead of `String.valueOf`.
- Add unit tests in `JdbcSinkFactoryTest` for `[]`, `[""]` and a list containing `null`.
**2. REST JSON submit endpoint.**
Confirmed: `ConfigBuilder.of(Map)` (`ConfigBuilder.java:107-110`) still uses `ConfigFactory.parseMap(objectMap)`, so the JSON body of the submit-job REST API rejects regex keys with `BadPath`. This PR documents it as a known limitation (`docs/en/connectors/sink/Jdbc.md`, and the zh counterpart), and the follow-up is tracked in #12434 ("REST submit-job with the JSON body rejects regex option keys"). The PR already fixes the file-based path and the `ConfigShadeUtils` rebuild through `rebuildWithLiteralKeys`, so the `of(Map)` fix would be a small extension of the same helper.
On whether it should block this PR: that is a scoping call. My view is that the documented limitation plus #12434 is acceptable, since config-file and HOCON-body submission work and the failure is explicit rather than silent. If you or the committers prefer to close the REST gap here, it is a one-line change in `ConfigBuilder.of(Map)`, but the helper would then need a test for the JSON-map path, so I would keep it in #12434 to keep this PR focused. I will defer to the committers on this one; point 1 is the part I consider a must-fix.
@luozihen, once the empty/null validation and its tests are pushed, I will re-review promptly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6355632069
Hi @hutiefang76 — thanks for the careful comparison. You are right that these are two different key contracts rather than two copies of the same fix, so it is worth agreeing on the contract and the ownership before either lands.
I ran the semantics against the shaded HOCON in this repo, because the thread mixes upstream typesafe-config behaviour with this fork's:
| input key | ConfigFactory.parseMap | literal rebuild in this PR |
| --- | --- | --- |
| job.mode, shade.identifier | literal key kept, hasPath(...) is true | identical |
| a->b | expanded to nested a -> b | one a->b key |
| " x " (quote characters in the key text) | normalized to ` x ` | kept as `" x "` |
| ^t_.*$ | ConfigException$BadPath | kept as a literal key |
The first row matters for the thread: this repo patches the shaded parser (#8975), so plain dotted keys are not expanded by parseMap here, and env.hasPath("shade.identifier") is true both before and after this change. I could not reproduce the REST shade consequence, but I agree it is exactly the kind of thing that should be pinned rather than argued, so I will add the REST-shaped test Daniel asked for (map -> RestUtil.buildConfig -> ConfigShadeUtils.decryptConfig, with a regex key, job.mode, shade.identifier and a->b).
The remaining deltas are the ones you named, and they are the contract question. Items I would like us to agree on:
1. Key contract for the shared conversion: all keys literal (current #12015), or quote only the keys that fail path parsing (#12443). I lean to the second: it keeps a->b and the quoting behaviour byte-compatible with today and only rescues keys that cannot be represented now.
2. Ownership: only one PR should change ConfigBuilder.of(Map), backfillUserVariables and ConfigShadeUtils.processConfig; the other drops its overlapping commit and rebases.
3. Scope: should ReadonlyConfig.toConfig(), the config-adapter path and the variable-map conversions (currently only in #12443) follow the same contract, or stay out of the shared change?
4. Merge order: whichever lands second removes its duplicate. 0c43f10dd here is a single, self-contained commit and easy to drop.
5. Tests that pin the contract: the REST-shaped case above, asserting which keys stay literal and which still expand.
Concretely, two ways to proceed:
- If you keep #12443 as the single shared fix with the quote-only contract, I will drop 0c43f10dd from #12015 and rebase onto it once it lands; the JDBC feature and its validation stay here.
- If the committers would rather have the REST path fixed in #12015 (@nzw921rx asked for that), I can switch of(Map) to quote-only-invalid here and you rebase #12443 on top, keeping the extra conversions (ReadonlyConfig.toConfig(), adapter, variable maps) in #12443.
Which of the two would you prefer? Once we agree on the contract and the owner, I will make the change the same day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6359807234
Thanks @luozihen for the thorough follow-up, and thanks @SEZ9 and @hutiefang76 for helping converge on the key contract. The head is still `d9bb1574f0dfee9f6673533dff2616522a76e6c4`, the one my latest review covers, so that review stands; this is a clarification, not a new full review. Static read only, nothing was executed locally.
**1. The first row of the semantics table does not match the shaded parser source on this head.** The table says `job.mode` / `shade.identifier` are kept as literal keys by `ConfigFactory.parseMap`. I traced the code path in this repo and it points the other way:
- `ConfigImpl.fromPathMap` (`seatunnel-config-shade/.../impl/ConfigImpl.java:189-193`) calls `PropertiesParser.fromPathMap`.
- `PropertiesParser.fromPathMap` (`PropertiesParser.java`, the map overload) builds `Path.newPath(key)` for every entry, and `Path.newPath` is `PathParser.parsePath` (`Path.java:233-235`), which splits on `.`.
- Nested map values are converted with `FromMapMode.KEYS_ARE_PATHS` (`PropertiesParser.java:175-177`), so a key inside `env` such as `shade.identifier` is also split into `shade` -> `identifier`. This is also why the regex key `^t_nova_.*$` fails with `BadPath` in the first place.
- The existing `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` (`ReadableConfigTest.java:383-390`) pins the same thing for `toConfig()`.
I could not find a patch to this path in `seatunnel-config-shade` (I searched for #8975 and found nothing), so unless you can point me at the exact class that changes this, I read the expansion as the current behavior. That is why I flagged Issue 1: with the literal rebuild in `0c43f10dd`, an `env` body carrying `shade.identifier` would no longer be found by `hasPath("shade.identifier")` in `ConfigShadeUtils`.
This is a factual question we can settle by test rather than by argument, and you already offered the right one. Please add the REST-shaped test exactly as you described (map -> `RestUtil.buildConfig` -> `ConfigShadeUtils.decryptConfig`) with `env` containing `shade.identifier`, `job.mode`, `a.b` and a regex key, and assert which keys still expand and which stay literal. If it is green on the literal rebuild, I will withdraw Issue 1 and say so; if it is red, it tells you which of the two options below to take.
**2. Which option to take.** Both are fine from my side. I agree with @SEZ9 and your own lean: keep existing keys behaving exactly as today and only rescue keys that cannot be parsed as a path today (quote-only-invalid). Only one PR should change `ConfigBuilder.of(Map)` / `ConfigShadeUtils.processConfig`; please agree the owner with @hutiefang76 and drop the duplicate commit in the other PR. If the owner is #12443, drop `0c43f10dd` here and restore the limitation wording in the two Jdbc docs; the JDBC feature and validation stay in this PR.
**3. CI, to correct one point in the thread.** The unit-test failure is not the `Open udf failed` log line (that is logged output). All four `unit-test` lanes fail on `LLMBooleanOutputTest.readsLegacyFactoryWrapperWithValidationDisabled` in `seatunnel-transforms-v2`, which this PR does not touch; the fix is #12938 (`cc54bb43`), and this head does not contain it (verified: it is not an ancestor of the head). So re-triggering alone will not turn it green; syncing the latest `dev` and re-running will. The dead link (the InterSystems IRIS driver URL) is on `dev` already and not in your diff, so I agree with @SEZ9 not to block on it.
Next step: sync `dev`, add the REST-shaped test, settle the owner of the shared conversion, then push; I will do a fresh pass on the new head. My earlier change request is still the visible state on this PR; Daniel has no write-level review controls, so once the blockers are cleared a maintainer with write access needs to do the final approve or dismiss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6212410642
@DanielLeens @SEZ9
CI is green now, and the blockers have been fixed. Could you please review it again? Please clearly point out what needs to be fixed before it can be merged. This PR has been open for too long. If possible, I’d like to make minimal changes and merge it first, then address the follow-up issues afterward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5582554227
@SEZ9
Thanks for the update, and for the clean checklist — that makes it much easier to confirm.
Here's the status against F1–F8 on the latest head:
**Addressed:**
- **F1** — I reverted `ReadonlyConfig#toConfig()` back to `ConfigFactory.parseMap(confData)`, so
the dotted-key path-expansion semantics are unchanged. The fix is now scoped to
`MultiTableFailureHelper.mergeOptions()`, which merges the option maps directly instead of
going through `toConfig()`.
- **F4** — `${primary_key}` / `${unique_key}` expansion happens only in the connector
(`expandPrimaryKeyPlaceholder`). The engine-level
`TablePlaceholderProcessor.replaceTablePlaceholder` only rewrites top-level String /
single-element-String-List values, so it leaves the nested `multi-table_config` map untouched —
no double processing.
**Addressed, but please confirm if you'd like it stricter:**
- **F2 / F5** — I kept the order-based "first match wins" behavior and documented it. This is backed by
the code path: `ReadonlyConfig.fromConfig` uses Jackson's `TypeReference<Map<String, Object>>`,
which deserializes to a `LinkedHashMap`, preserving declaration order; I did not add a separate
order-preservation test, so if you'd prefer I pin it down with a test or an explicit `LinkedHashMap`
type, I can do that.
**Still open — happy to address as follow-ups:**
- **F3** — I haven't added explicit identifier validation for the resolved key columns; they are
still comma-joined into `PRIMARY_KEYS` and rely on the existing list conversion.
- **F6** — regex patterns are currently validated lazily at match time (thrown as `JDBC-12`), not
up-front during config validation.
- **F7** — I haven't added a regression test for the restored `toConfig()` dotted-key behavior, nor
an engine-level IT/E2E for the new option (only unit tests so far).
- **F8** — the key is `multi-table_config` to match the name we settled on in the issue. If the
project convention prefers a different form, I'm fine renaming it.
Happy to take the open ones on, or wait for your targeted comments and fix them accordingly.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5612023677
@DanielLeens
Thanks for taking the trouble to review this again. Regarding the latest issues raised, I have submitted another commit that caches the configured patterns so they don’t have to be parsed separately for each table. I’ll let you know once CI turns green.
@SEZ9
Thank you for being so patient and for providing such detailed suggestions on each issue. However, when I look at the truncated issues you are now raising / previously mentioned, they don’t seem to exist on my side. Could this be a frontend rendering issue? I’m not sure, because from what I can see, everyone’s comments appear complete on my end. In addition, regarding the latest suggestions from @DanielLeens, I have already updated accordingly; you can review based on the latest commit. Sorry for the trouble, and thank you both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5488587749
@luozihen thanks for flagging this proactively instead of just leaving CI red — that's exactly the right instinct, and this kind of accidental merge-into-feature-branch mistake is extremely common, so no need to apologize for the noise!
My preference here is actually **option 2 (rebase), not option 1 (revert)** — and I'd suggest an even simpler variant than a manual interactive rebase:
```bash
git fetch origin dev
git rebase origin/dev
# resolve any real conflicts if your own commits touch lines dev has since changed
git push --force-with-lease
```
A plain (non-interactive) `git rebase origin/dev` will automatically drop the merge commit for you — rebase replays only the commits that are yours (not already reachable from `origin/dev`), and a merge commit whose "left" parent is already on `origin/dev` contributes no new patch of its own, so it gets skipped rather than replayed. You shouldn't need to hand-pick which commit to drop.
Why rebase over revert in this specific case:
- A revert commit cancels out the upstream changes in your branch's history, but it doesn't remove the merge commit itself — your branch still carries an extra merge commit *and* a revert commit, both of which are irrelevant to this PR's actual diff. That's more history noise, not less, and it can make the "Files changed" tab harder to read while this PR is under review.
- Force-pushing here is safe: this is your own PR branch, nobody else has based work on top of it, and Apache SeaTunnel squash-merges PRs on landing anyway, so the interim commit history won't survive into `dev` either way. There's no shared-history risk that the usual "don't force-push" caution is meant to protect against.
- A clean rebase means CI runs against exactly your feature diff on top of current `dev`, with nothing from a stale merge snapshot mixed in — which also makes it easier for me to confirm the green run actually reflects your changes when I re-review.
Go ahead with the rebase + force-push, and ping me once CI is green again — I'll take another look. Thanks again for the careful contribution and for keeping the PR history clean, that's a great habit to build early!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673722746
@SEZ9
F4 — Fail-fast path: `JdbcSinkFactory#expandPrimaryKeyPlaceholder` throws
`JdbcConnectorException(INVALID_MULTI_TABLE_CONFIG / JDBC-12)` when `${primary_key}` or
`${unique_key}` is used but the upstream table has no such key. Test:
`JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysMissingPrimaryKeyFails`.
The docs already state this: `docs/en/connectors/sink/Jdbc.md:316` and
`docs/zh/connectors/sink/Jdbc.md:316`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673732290
@SEZ9 F7 — The regression test is `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion`. The
deferred engine-level IT/E2E is tracked in the PR description under a "Follow-up" section:
"An engine-level IT/E2E for `multi_table_config.primary_keys` is deferred and will be tracked
separately; the current coverage is unit-test level."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6406899455
Hi @luozihen, @hutiefang76, @SEZ9, thanks for pushing this to a concrete answer, and @luozihen, I owe you a correction.
I re-checked the shaded parser on `dev`: `ConfigParseOptions.PATH_TOKEN_SEPARATOR` is `"->"`, and `PathParser.splitTokenOnPeriod` / `addPathText` split only on that token. So in this repository a plain dotted key such as `shade.identifier` or `job.mode` is not expanded by `parseMap`; it stays one literal key, and `env.hasPath("shade.identifier")` is true both before and after the literal rebuild. That means the premise of my earlier "Issue 1 (High)" (that `0c43f10dd` stops `env`'s dotted shade keys from being found) was wrong, and my +1 on the expansion concern was wrong for the same reason. Sorry for the detour. I also misread `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion`: it only asserts `getInt("a.b")`, which passes for a literal key too, so it does not pin expansion.
What does still differ, per your table, is narrower: `a->b` (expands today, would become one literal key) and keys containing quote characters (normalized today, kept as-is). Those can matter for existing REST JSON users, so I agree with the plan you described: quote only the keys that fail path parsing and leave everything that parses today untouched, with the REST-shaped case (`RestUtil.buildConfig` -> `ConfigShadeUtils.decryptConfig`) asserting `shade.identifier`, `job.mode`, `a.b`, `a->b` and a regex key. With that in place I will re-review the new head and withdraw the compatibility blocker. Please also fix the comment in `ConfigBuilderTest` that describes the "before" shape.
Two practical notes: please settle ownership of `ConfigBuilder.of(Map)` / `ConfigShadeUtils` with @hutiefang76, since #12443 already carries the quote-only-invalid helper and its `Build` is green on its current head (I am approving it separately); reusing that helper, or rebasing on it, avoids two competing conversions. And the red `Build` here is the `LLMBooleanOutputTest` failure in `seatunnel-transforms-v2`, fixed on `dev` by #12938 and absent from this branch, which is 33 commits behind, so syncing `dev` with your next push will clear it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6356176698
@luozihen thanks for digging through the CI logs.
**Issue 1 (dead link):** I cannot tell from the thread whether that URL comes from the base branch or from the JDBC sink doc this PR touches. Could you check whether the link appears in your diff? If it is only in the base branch, a rebase onto the latest main should tell us, and a docs-only fix in a separate PR would be the right place for it; I would not block this PR on it. If it is in your diff, please update or remove the link here.
**Issue 2 (`ZetaSQLEngine - Open udf failed` in unit-test):** re-triggering is fine; I do not expect a code change from you for it unless it reproduces consistently.
On the `ReadonlyConfig.toConfig()` key-contract thread, a few concrete asks to close it out:
1. I lean the same way you do: quote only the keys that fail path parsing, so existing keys keep their current behaviour and only keys that cannot be represented today are rescued.
2. Only one PR should carry the shared conversion change. If it lands elsewhere, please drop `0c43f10dd` from this PR and rebase once that is merged; if it lands here, keep it as the single self-contained commit it is now.
3. Please add the REST-shaped test you described (map -> `RestUtil.buildConfig` -> `ConfigShadeUtils.decryptConfig` with a regex key, `job.mode`, `shade.identifier` and `a->b`), asserting which keys stay literal and which still expand. Since that exercises the REST path rather than `ReadonlyConfig.toConfig()` directly, please also add a direct `toConfig()` test for the dotted-key behaviour so both paths are pinned.
The JDBC-specific items from the earlier review (declaration-order guarantee for the map-typed option and its docs, identifier validation on the resolved key columns, the `${primary_key}`/`${unique_key}` vs. `TablePlaceholder` interaction, regex compile-time validation mapped to JDBC-12, and the `multi-table_config` naming) are independent of the shared conversion, so feel free to push those regardless of which PR carries the config change.
<!-- streview-comment:1963 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6137986285
Thanks for keeping the branch in sync with `dev`. Since the only new commit on this head is the merge (`e761f0c5b`) and the PR-owned files are reported byte-identical to `00acc23a5`, my earlier review points stand as-is. Here's where each one lands and what I still need:
**F1 – `ReadonlyConfig.toConfig()` dotted-key semantics (HIGH).** The latest re-review states that `ReadonlyConfig#toConfig()` is no longer part of the diff and that the regex-key handling now lives in `ConfigShadeUtils.processConfig` and `MultiTableFailureHelper.mergeOptions` instead. If that holds, this resolves my main compatibility concern, and dotted-key expansion for all other callers stays as before.
**F2 / F5 – "first pattern in declaration order wins" (HIGH / MEDIUM).** Still open. The option is a `Map<String, Object>`, so declaration order is not guaranteed unless the map is explicitly order-preserving end to end (HOCON parse → `ReadonlyConfig` → `resolveMultiTablePrimaryKeys`). Please either (a) show where an insertion-ordered map is enforced and add a unit test with two overlapping patterns asserting the first one wins, or (b) change the contract (e.g. reject overlapping matches, or document that order is undefined) in both `JdbcSinkOptions` and `docs/en/connectors/sink/Jdbc.md`.
**F3 – resolved key columns flow into generated SQL (MEDIUM).** Still open. Before the comma-joined value is written into `PRIMARY_KEYS`, please validate each column against the upstream table schema (or at minimum a strict identifier pattern) and fail with a clear error instead of letting arbitrary strings reach SQL generation.
**F4 – connector-level `${primary_key}` / `${unique_key}` vs engine-level `TablePlaceholderProcessor` (MEDIUM).** The re-traced runtime path (engine placeholders first, then connector expansion) is helpful. Please add a short note to the docs describing this ordering and what happens when the upstream table has no PK/unique key (empty expansion vs. error), plus a test covering that case.
**F6 – regex compile validation (MEDIUM).** Please make sure an invalid pattern fails at `createSink` time with the new JDBC-12 error code (not a raw regex exception from the cache), and add a unit test for it.
**F7 – tests (MEDIUM).** If F1 is indeed resolved by not touching `ReadonlyConfig`, the regression test there is no longer required. Still needed: an IT/E2E running a real multi-table JDBC sink job that exercises `multi_table_config.primary_keys` with at least one matched and one unmatched table.
**F8 – option naming (MEDIUM).** The latest text refers to the option as `multi_table_config.primary_keys` (snake_case). Please confirm the actual key in `JdbcSinkOptions` and the docs match that spelling and that the earlier `multi-table_config` mixed form is gone everywhere.
Once F2/F5, F3, F6 and the IT from F7 are addressed, I'm happy to take another full pass.
<!-- streview-comment:1723 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6282033499
Thanks @luozihen for the quick follow-up. I see `82fc5e61a` (on top of `231cbdea5`) adds `JdbcMultiTableConfigConditionExtension` so that a mis-shaped `multi_table_config` fails during option validation instead of silently falling back — that closes the fail-fast concern.
Before approving, I'd like to confirm the status of the other points from the earlier round. Could you reply to each inline, or point me at the commit/test that covers it?
1. **`ReadonlyConfig.toConfig()` dotted-key round-trip** — this lives in `seatunnel-api`, so it affects every connector, not only the JDBC sink. What is the behaviour for existing callers that relied on dotted keys being expanded into paths? If the semantics changed, please spell it out and cover it with a test.
2. **"First pattern in declaration order wins"** — the option is a `Map<String, Object>`, and neither that map nor HOCON object keys guarantee declaration order. Either use an order-preserving structure that is actually populated in declaration order (and assert it in a test), or adjust the en/zh docs to not promise this.
3. **Resolved key columns flowing into generated SQL** — the matched columns are comma-joined into `PRIMARY_KEYS` and end up in `generate_sink_sql` output. Please add identifier validation (or dialect quoting) so a bad regex value can't produce broken or unsafe SQL.
4. **`${primary_key}` / `${unique_key}` vs. engine `TablePlaceholder`** — please document (code comment and PR description) which replacement runs first and what happens when both apply to the same value.
5. **Invalid regex** — does the new condition extension also compile each pattern key and map a regex syntax error to the new JDBC-12 error code, or is that still only hit lazily at match time? A unit test with an invalid pattern would settle it.
6. **Tests** — I still don't see a regression test in `ReadableConfigTest` for the previous dotted-key expansion, nor an engine-level IT/E2E exercising `multi_table_config.primary_keys` end to end. Both are needed for a change that touches `seatunnel-api`.
7. **Option key naming** — the current head reads `multi_table_config` (snake_case), matching the JDBC sink convention. Please confirm the hyphenated `multi-table_config` spelling is gone from both code and docs.
Once those are answered I'll do a final pass on the current head.
<!-- streview-comment:1887 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4311851250
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
Agreed on fail-fast. Please add the structural check in optionRule() via Conditions.extension(...), same pattern as `JdbcTableOptionsConditionExtension`
```java
.optional(
JdbcSinkOptions.MULTI_TABLE_CONFIG,
Conditions.extension(
JdbcSinkOptions.MULTI_TABLE_CONFIG,
new MultiTableConfigValidator()))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4310580048
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
I understand that configuring `JdbcSinkOptions.MULTI_TABLE_CONFIG `indicates that the user has an explicit customization request. However, when `primary_keys` is not configured, the current implementation silently falls back. I think this case should fail fast instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6096399262
@SEZ9 Thanks for your help. The MinIO issue was resolved after I merged the latest code. Now I’m seeing an issue with the elasticsearch-connector-it tests. I noticed that the submitter of another PR (#12708) is also asking about this. Could you please take a look as well and help us figure out how to handle it, so that PRs unrelated to ES can pass normally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5578481706
@luozihen Congratulations on getting CI all the way to green, and welcome to the Apache SeaTunnel community — this has been a genuinely solid first contribution, and the way you dug into the HOCON/CI issues yourself along the way was great to see.
Where things stand right now, for your reference: I approved this PR on my side after the last round of fixes, and the latest run on `1730aeef` is fully green (Build, labeler, and the notify workflow all completed successfully).
On next steps, here is roughly how it works in an Apache project like this one:
1. **My approval alone isn't enough to merge.** Apache SeaTunnel requires review/approval from a project committer with merge rights before a PR can go in — my account here doesn't hold that permission, so a second maintainer still needs to look this over. That's normal and not a sign anything is wrong with your PR.
2. **@SEZ9 is doing exactly that second pass** — they've already asked for a short checklist confirming which items from the earlier review round your last push addressed. Please answer that when you have a moment; it'll let them wrap their confirmation quickly.
3. **Timeline**: since SeaTunnel is a volunteer-driven Apache project, maintainer availability varies — it can take anywhere from a few days to a couple of weeks to get a second reviewer's attention, especially for a larger feature PR like this one. There's nothing further you need to do to speed this up beyond answering open review questions promptly; no need to re-ping unless it's been quiet for a while.
4. Once a maintainer with merge rights approves, they (or another committer) will merge it — normally as a squash merge onto `dev`.
Thanks again for your patience through the CI back-and-forth, and for the clear, cooperative communication throughout review — that makes this a lot smoother for everyone. Nicely done for a first PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5569399709
@luozihen Good question — and yes, this is still the same unrelated Doris-connector flake I called out in my last review, not a new failure. I re-checked the fork run just now (`luozihen/seatunnel` run `33743266792`, head `1730aeef`): the only two non-passing jobs are `doris-connector-it (8, ubuntu-latest)` and `doris-connector-it (11, ubuntu-latest)`, both Testcontainers/Docker networking symptoms against the Doris FE/BE containers, plus `kudu-connector-it` which was `cancelled` (a downstream side-effect of the matrix, not a real failure) and a batch of `updated-modules-integration-test-*` jobs that were `skipped`. Nothing in the JDBC/config modules this PR touches is failing.
Since this PR's CI runs inside your own fork, you have direct rerun access without needing me or another maintainer:
- UI: open the run page on `luozihen/seatunnel` (Actions tab -> this run -> "Re-run jobs" -> "Re-run failed jobs", top-right).
- CLI (with your own GitHub auth, not mine): `gh run rerun 33743266792 --repo luozihen/seatunnel --failed`
That reruns only the failed/skipped legs, not the whole matrix, so it should be quick. To be clear: CI hasn't actually re-executed since your last push (the HOCON fix + rebase) — GitHub is just still displaying that same red run from 2026-09-03. A rerun is exactly the right next step here, no further code change needed on your side. If it comes back green (or fails again with the same Doris/Testcontainers signature), ping this thread and I'll do a fresh confirmation pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5564702401
@DanielLeens
Hi, could you please advise on how to rerun only the failed tests? The CI has been consistently red, and I'm wondering what I should do next. Any guidance would be appreciated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5613705069
@DanielLeens
Thanks for the careful catch on the static cache — that's a fair trade-off to call out.
For Issue 1, I’m currently leaning toward the bounded-LRU option rather than the per-job scoping you preferred, but I haven’t finished that change yet. Here’s the reasoning:
The connector's JdbcSinkFactory.createSink() is invoked once per table by the engine's per-table loop (MultipleTableJobConfigParser), not once per job, and the TableSinkFactory contract has no per-job initialization hook where I could compile the patterns once and pass the compiled map down through that loop. Doing that would require touching the engine's sink-planning path, which feels outside the scope of this connector PR.
So, to keep the change local while still bounding the memory, I’m planning to replace the unbounded ConcurrentHashMap with a small, JDK-only bounded LRU (LinkedHashMap with removeEldestEntry, wrapped in Collections.synchronizedMap, capped at 256 entries). This should keep the memoization benefit (each distinct pattern compiles once and is reused) while putting a hard upper bound on growth, without introducing a new dependency such as Caffeine. I also plan to add a short comment noting that the cache is shared across jobs in this JVM and is bounded.
I’ll also note the engine-level IT/E2E as a follow-up in the PR description, as agreed.
If you'd still prefer the per-job scoping, I'm happy to follow up separately — I just wanted to avoid expanding the change surface into the engine within this PR. Let me know what you'd prefer. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5724793526
Thanks for the thorough pass — seven of these eight items were already addressed in the earlier rounds, so instead of re-touching working code I've pointed each one at the current source, and added the one thing that was genuinely missing (a test for the placeholder pass ordering in F4). Everything below is on the new head:C15F9FD.
F1 — ReadonlyConfig#toConfig() compatibility. toConfig() is unchanged: it is still ConfigFactory.parseMap(confData) (seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java:76), and ReadonlyConfig.java is not part of this PR's diff at all. The only config-rebuild change here is in ConfigShadeUtils.processConfig, which replaces ConfigFactory.parseMap(configMap) with a JSON round-trip; without it a regex key such as ^t_nova_.*$ is re-read as a HOCON path and the job fails with ConfigException$BadPath before the sink ever sees the option. The map being re-encoded holds the values of an already-parsed config, so JSON re-encoding keeps each key literal rather than applying path expansion a second time. MultiTableFailureHelper#mergeOptions() no longer calls toConfig()/withFallback() either — it merges the two option maps directly, with the shallow-merge contract documented on the method. The previous dotted-key expansion of toConfig() is pinned by Read
ableConfigTest#testToConfigPreservesDottedKeyExpansion.
F7 — tests. Both halves exist. ReadableConfigTest#testToConfigPreservesDottedKeyExpansion asserts that an a.b key still expands through the dotted path. The engine-level E2E is JdbcMysqlMultipleTablesIT#testMysqlJdbcMultipleTableE2e with jdbc_mysql_source_and_sink_with_multi_table_config.conf: it drops the sink tables, lets the sink auto-create them, then asserts the resolved per-table primary keys and row counts, and it runs on the Zeta/Flink/Spark containers. It uses the JDBC source rather than MySQL-CDC deliberately — everything this option touches is resolved sink-side in createSink(), so the source connector type does not change the path under test. If you would like the CDC-sourced variant as well I can add it, but I would suggest tracking it as a follow-up, since another @TestTemplate in the CDC suite adds a full container x JDK matrix run.
F2 / F5 — ordering. toCompiledPatternMap copies the configured patterns into an explicit LinkedHashMap<Pattern, List<String>>, so ordering is guaranteed by construction rather than by whatever map the parser hands over; JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins parses a real HOCON string whose patterns overlap in non-alphabetical declaration order (^table2$, ^table1$, then a catch-all ^table.*$ that would win if the order were lost) and asserts the first declared match wins. The en/zh docs state the same rule. I kept declaration-order semantics rather than "fail fast when more than one pattern matches", because overlapping patterns are the intended usage here (a broad default plus narrower overrides).
F3 — identifier safety. validatePrimaryKeyColumns rejects blank names and names containing a comma before they are joined into PRIMARY_KEYS, raising JDBC-12, covered by JdbcSinkFactoryTest#testFactoryContextWithMultiTableConfigInvalidColumnFails. Those two are the only inputs that cannot be represented safely: the resolved names are interpolated through the dialect's quoteIdentifier when SQL is generated (JdbcDialect#getDeleteStatement, MysqlDialect#quoteIdentifier, MysqlCreateTableSqlBuilder#buildPrimaryKeySql), so quoting covers everything else, while a comma collides with the delimiter of the comma-joined primary_keys value and a blank name produces an empty list element. That reasoning is now recorded on the method's Javadoc.
F4 — pass order. Documented and tested. docs/{en,zh}/introduction/configuration/sink-options-placeholders.md state the fixed order explicitly, and JdbcSinkFactoryTest#testPlaceholderOrderBetweenEngineAndConnectorPasses pins it: it runs the real engine pass first and asserts that a top-level primary_keys = ["${primary_key}"] is expanded there, that the nested multi_table_config.primary_keys map is still untouched afterwards, and that the connector expands it during createSink().
F6 — regex handling. Patterns are compiled once in the factory (compilePattern, memoized in the bounded COMPILED_PATTERN_CACHE, 256 entries) and an invalid pattern raises JDBC-12 with the offending pattern in the message; covered by testResolveMultiTablePrimaryKeysInvalidRegexFails, testCompilePatternInvalidFails and testCompilePatternCachesCompiledPattern.
F8 — naming. Confirmed multi_table_config in JdbcSinkOptions, both docs, the E2E config and all tests; the last hyphenated dummy key in ConfigShadeTest was renamed as well.
One item from the previous round that is not on your list, but worth knowing: 6043c1507 also fixed the name the patterns are matched against. createSink() previously matched against the already-remapped sink table name, so combining this option with table / tablePrefix / tableSuffix silently defeated the mapping and fell back to catalog metadata; it now keeps the upstream CatalogTable and matches on that, with JdbcSinkFactoryTest#testFactoryContextWithMultiTableConfigMatchesUpstreamTableName as the regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5770190113
@luozihen Yes — I'll treat the "Review status (F1-F8)" section of the PR description as the source of truth for the next pass. Note that this comment was cut off mid-F4 on my side too, so everything from F4 onward I can only take from the description.
On what did come through:
- **F1** — If `ReadonlyConfig.java` is not in the diff and `toConfig()` is unchanged, the compatibility concern as originally written does not apply, so I'm fine dropping it as a `ReadonlyConfig` issue. One remaining ask: please confirm (in the F1 entry) whether the parsing change in `ConfigShadeUtils.processConfig` / `MultiTableFailureHelper#mergeOptions` is scoped to the new multi-table option only, or whether it alters how dotted keys are parsed for any other option that flows through those two methods.
- **F2 / F5** — `LinkedHashMap<Pattern, List<String>>` in `toCompiledPatternMap` guarantees the compiled map preserves whatever order it receives; the real question is the order of the input map, which is exactly what `testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins` needs to prove. Please make sure that test goes through the same path the runtime uses (HOCON string → `ReadonlyConfig` → option → `toCompiledPatternMap`) rather than a hand-built map. If it does, F2 and F5 are resolved for me.
- **F3** — Rejecting blank names and commas with `JDBC-12` plus dialect quoting covers the cases I raised. Only follow-up: does the dialect quoting also escape a quote character appearing inside a column name, or should `validatePrimaryKeyColumns` reject those as well? Either answer is fine; I just want it to be deliberate.
For **F4–F8** I'll read the description, but to save a round trip please make sure those entries name: the test for the `${primary_key}` / `${unique_key}` vs. engine-level placeholder ordering (F4), where invalid regex patterns are compiled and mapped to `JDBC-12` (F6), the regression test and the IT/E2E for the new option (F7), and the final decision on the `multi-table_config` key name (F8).
<!-- streview-comment:1218 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5770455151
Thanks for the very thorough pass — the log analysis, the CI breakdown and the four carryover findings are all clear. Here is how I handled each one; the changes are in 00acc23a582d76a589c00df956dded4b6f21da94.
- Issue 1 (regex keys on the REST JSON path) — handled as your option A.
You are right that the JSON body bypasses the fix: RestUtil.buildConfig / buildConfigList turn it into a map and hand it to ConfigBuilder.of(Map) before the shade rebuild, so ConfigFactory.parseMap(...) still rejects ^t_nova_.*$ with ConfigException.BadPath. To keep this PR focused I did not change that path here: I prototyped option B, verified it, and then reverted it so the diff stays scoped. Instead the limitation is now documented in the multi_table_config section of docs/en/connectors/sink/Jdbc.md and docs/zh/connectors/sink/Jdbc.md — regex keys work for config files and for the HOCON REST body, the JSON REST body is not supported yet and rejects the configuration while parsing it. I will open a follow-up issue for ConfigBuilder.of(Map) (rebuild the map with literal keys when a key is not a valid path, plus a test for the REST-shaped map) and link it here.
For completeness: that change would only affect the of(Map) overload, which has exactly two production callers, both on the REST submit path (RestUtil.buildConfig / buildConfigList); every other ConfigBuilder.of(...) caller uses the Path/String overloads that this PR does not touch.
- Issue 2 (legacy primary_keys branch) — fixed as you suggested.
applyFallbackPrimaryKeys now rebuilds only the catalog primary key for the explicit top-level primary_keys case and leaves the option value untouched, exactly as on dev; the column validation and the comma-joined write happen only on the multi_table_config branch, where the value is computed. The shared catalog rebuild lives in buildCatalogTableWithPrimaryKey, so the two branches cannot drift. Guarded by JdbcSinkFactoryTest#testLegacyPrimaryKeysStillRebuildCatalogPrimaryKey.
- Issue 3 (PR title) — fixed, the title now says multi_table_config.
- Issue 4 (test for the processConfig round trip) — added.
ConfigShadeTest#testDecryptPreservesOrdinaryConfigShapes covers an unquoted dotted key, a quoted dotted key inside a nested map and numeric values, and asserts that the env, source and sink blocks come back unchanged. I compare those blocks instead of the whole config because processConfig always writes an empty transform key (pre-existing behaviour), which would make a whole-config equality assertion fail for a reason unrelated to the round trip.
- Issue 5 (mistyped static column) — left as a follow-up, as you suggested: validating names against the schema has to account for field_ide case conversion, so it is cleaner as its own change. It is recorded in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673720538
@SEZ9 F3 — Identifier validation is in place: `JdbcSinkFactory#validatePrimaryKeyColumns` (called from the
shared `applyPrimaryKeys`) rejects blank names and names containing a comma before they are
comma-joined into `PRIMARY_KEYS`. Test:
`JdbcSinkFactoryTest#testFactoryContextWithMultiTableConfigInvalidColumnFails`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673712967
@SEZ9
Thanks for the follow-up. Since long replies keep getting truncated, I'll post the per-item status as
separate comments (F1–F8) so nothing gets cut off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5673718729
@SEZ9
F2 — Order is guaranteed by an explicit `LinkedHashMap`: `toCompiledPatternMap` builds
`LinkedHashMap<Pattern, List<String>>` from the config map, and upstream `ReadonlyConfig.fromConfig`
deserializes via Jackson `TypeReference<Map<String, Object>>`, which yields a `LinkedHashMap`, so
HOCON declaration order is preserved end to end. Test:
`JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFirstMatchWins`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5691251373
@SEZ9 Since long comments kept getting truncated, I've moved the per-item status for F1–F8 into the PR
description (see "Review status (F1–F8)"). Please take a look there when you do the next pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5846333267
Thanks for narrowing it down to these four, @SEZ9 — I went and re-checked each one directly against the source at `00acc23a582` rather than relying on the description prose.
**F1** — Confirmed unchanged. `ReadonlyConfig#toConfig()` (`ReadonlyConfig.java:76`) is still exactly `ConfigFactory.parseMap(confData)`. The two actual changes are:
- `ConfigShadeUtils.processConfig` (`ConfigShadeUtils.java:224-226`) now rebuilds the post-decrypt map with `ConfigFactory.parseString(JsonUtils.toJsonString(configMap), ConfigParseOptions.defaults().setSyntax(ConfigSyntax.JSON))` instead of `ConfigFactory.parseMap(configMap)`. Since `configMap` is already the result of a JSON parse, this keeps every key literal on the round trip instead of re-interpreting it as a HOCON path — that is what lets a regex key such as `^t_nova_.*$` survive the encrypt/decrypt step. It runs for every job, but it only changes behavior for keys that previously failed to parse as a HOCON path (or that relied on being re-expanded as one at this specific point); I don't see a caller in this repo that does the latter.
- `MultiTableFailureHelper.mergeOptions()` (`MultiTableFailureHelper.java:64-71`) no longer goes through `Config#withFallback()`. It now does `merged.putAll(fallback.getSourceMap()); merged.putAll(primary.getSourceMap())` and wraps the result with `ReadonlyConfig.fromMap()`. The Javadoc directly above it now states: "The merge is shallow/top-level: on a key collision, primary's value replaces fallback's value wholesale, and nested objects are not merged recursively." That's the exact caveat we pinned down a few rounds back, now written into the code rather than left implicit.
So the compatibility impact on other `toConfig()` callers is nil — that method never changed. The only shared-code shift is `mergeOptions()`'s merge strategy, and every current caller (`MultipleTableJobConfigParser`, the Spark/Flink `SinkExecuteProcessor`s, `withMultiTableFailurePolicy`/`withFailedTables`) merges disjoint namespaces, so the shallow-vs-deep distinction is a no-op for them today.
**F2/F5** — This is exactly covered end to end. `JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins` (`JdbcSinkFactoryTest.java:348-380`) parses a real HOCON string containing:
```
"^table2$" = ["c_mediumint"]
"^table1$" = ["c_int", "c_integer"]
"^table.*$" = ["c_smallint"]
```
declared in that non-alphabetical order — the catch-all `^table.*$` is declared last even though it would sort first — builds a `ReadonlyConfig` via `ReadonlyConfig.fromConfig(...)` from the parsed `Config`, and calls the production `factory.resolveMultiTablePrimaryKeys(...)` directly, not a hand-built map. It asserts `table1` resolves to `["c_int", "c_integer"]` and `table2` to `["c_mediumint"]` — the first-declared pattern wins for both, not the catch-all. That is the full chain you asked for (HOCON string → `ReadonlyConfig` → option → `toCompiledPatternMap`), so I'd consider F2/F5 resolved.
**F4** — Also written down now. `docs/en/introduction/configuration/sink-options-placeholders.md:145-148` (and the `docs/zh` counterpart) states: "The two passes run in a fixed order. The engine-level replacement described above runs first and only [rewrites top-level String/single-element-String-List values, so the] `multi_table_config.primary_keys` map reaches the sink untouched. The JDBC sink then expands `${primary_key}` / `${unique_key}` inside that map once per matched table, while the sink is created." That matches what both of us independently traced through `TablePlaceholderProcessor`/`expandPrimaryKeyPlaceholder` earlier in this thread.
**F7** — `ReadableConfigTest#testToConfigPreservesDottedKeyExpansion` (`ReadableConfigTest.java:383-391`) builds a map with a single dotted key `a.b`, round-trips it through `ReadonlyConfig.fromMap(...).toConfig()`, and asserts `config.getInt("a.b") == 1` — the old HOCON path-expansion behavior for dotted keys is intact for every caller of `toConfig()`.
`JdbcMysqlMultipleTablesIT#testMysqlJdbcMultipleTableE2e` runs the actual `multi_table_config.primary_keys` scenario end to end through the engine: it drops the dedicated sink tables so the sink has to create them itself, executes `jdbc_mysql_source_and_sink_with_multi_table_config.conf`, and then asserts the primary keys read back from the database catalog (not from the job's own config) — `mtc_table1` → `["c_int", "c_integer"]`, `mtc_table2` → `["c_mediumint"]` — with a comment noting the catch-all pattern is declared last on purpose. So this also re-proves first-match-wins at the engine level, not only in the factory unit tests.
With F1, F2/F5, F4 and F7 now independently verified against the diff, and F3/F6/F8 already confirmed in my previous pass, I don't have anything further open on my side. Ready to merge from my read; still waiting on a write-capable maintainer for the actual merge action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6207000332
Thanks for keeping the branch in sync with `dev`. To be clear up front: my earlier "ready to merge" on `e761f0c5b` does not carry over to the current head (`7340000b87abd01130fdaaa4623f5e6a572e6d88`), so I re-reviewed it from scratch. The PR-owned files are unchanged, and the feature logic itself still traces cleanly (declaration order is preserved via the `LinkedHashMap` of compiled patterns).
What changed is `dev`: the merge pulled in a change that reintroduces the exact `BadPath` failure this PR exists to fix, on a code path the PR does not cover. On this head, regex keys such as `^t_nova_.*$` no longer survive config loading. This is a blocking regression. Could you extend the fix to that path and add a test that loads a job config with a regex-keyed `multi_table_config.primary_keys` map end-to-end (ideally an engine-level IT/E2E with at least two matching tables and one unmatched fallback table)?
Separately, a few smaller carryover items I should have raised earlier, apologies:
- Resolved key columns are comma-joined into `PRIMARY_KEYS` and reach generated SQL; please validate each column name and raise `INVALID_MULTI_TABLE_CONFIG` on failure.
- Please make an invalid regex fail fast at `createSink` time with the JDBC-12 error code and the offending pattern in the message, with a unit test.
- Please add a short note (code comment and docs) on which layer expands `${primary_key}` / `${unique_key}` when the engine-level placeholder replacement is also in play, plus a unit test mixing a placeholder with a static column such as `DATA_SOURCE`.
Once the config-loading regression is covered I'm happy to do a final pass.
<!-- streview-comment:1787 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5724181306
Thanks for the quick follow-up in `6043c1507`. Capturing the upstream `CatalogTable` in `JdbcSinkFactory.createSink()` before the variable is reassigned to the sink-side table, and passing it into `resolveMultiTablePrimaryKeys`, is the minimal fix I was hoping for — the regex now matches on the upstream table name regardless of `table`/`tablePrefix`/`tableSuffix` remapping, which is what the docs already describe. Good to see a regression test alongside it.
On the earlier review points, here is what I still need before merging:
- **PR12015-F1 (Compatibility, `ReadonlyConfig.toConfig()`)** — the JSON round-trip changes how dotted keys are interpreted for every caller of `toConfig()`, not only the JDBC sink. Please either scope the change so only the new map option is affected, or explain why dropping the old dotted-key path expansion is safe for all callers.
- **PR12015-F7 (Test)** — please add a test in `ReadableConfigTest` that pins the previous dotted-key behavior of `toConfig()`, plus an engine-level IT/E2E exercising `multi_table_config.primary_keys` end to end (ideally the MySQL-CDC -> JDBC `generate_sink_sql = true` scenario this PR targets).
- **PR12015-F2 / PR12015-F5 (ordering)** — the docs promise "first pattern in declaration order wins", but the option is a `Map<String, Object>` and HOCON object keys don't guarantee declaration order. Either switch to an order-preserving structure and add a test proving the order survives parsing, or change the docs and fail fast when more than one pattern matches a table.
- **PR12015-F3 (Security)** — the resolved key columns are comma-joined into `PRIMARY_KEYS` and end up in generated SQL. Please validate each column name as an identifier (or quote it via the dialect) before it reaches SQL generation.
- **PR12015-F4 (Functional)** — please document (and test) the order between the connector-level `${primary_key}`/`${unique_key}` expansion and the engine-level TablePlaceholder replacement.
- **PR12015-F6 (Robustness)** — compile the user-supplied regex patterns once at factory time and map invalid patterns to the new JDBC-12 error code, including the offending pattern in the message.
- **PR12015-F8 (Style)** — I flagged the key `multi-table_config`; the latest description refers to `multi_table_config`. Please confirm the key is `multi_table_config` consistently in `JdbcSinkOptions`, the docs, and example configs.
If any of these were already handled in `6043c1507`, just point me at the spot and I'll take a look.
<!-- streview-comment:1127 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5852053274
Thanks for re-checking each point against `00acc23a582`. Here is where things stand from my side.
**F1 (Compatibility)** — Agreed: with `ReadonlyConfig#toConfig()` still being `ConfigFactory.parseMap(confData)`, the finding as originally worded no longer applies. Two follow-ups on the places that did change:
- `ConfigShadeUtils.processConfig` rebuilding the post-decrypt map via a JSON round trip: the reasoning that `configMap` is already the output of a JSON parse (so keys stay literal) sounds right. Since this runs for every job, could you add a small regression test on the shade path asserting that a config with a dotted key and a nested object both come out of `processConfig` unchanged, alongside the positive case for a regex key such as `^t_nova_.*$`? That would also cover the toConfig/shade half of F7.
- `MultiTableFailureHelper.mergeOptions()` switching to a shallow top-level `putAll` merge is fine given the Javadoc now states it explicitly and current callers merge disjoint namespaces. No further ask here.
**F2/F5 (declaration order)** — Your comment appears to cut off right after describing the three-pattern HOCON fixture in `testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins`, so I can't see what the test asserts. Could you post the remainder? Specifically I'd like to confirm that (a) the exact-match pattern wins over the catch-all declared last, i.e. non-alphabetical declaration order is honored, and (b) the order guarantee holds on the option's underlying `Map<String, Object>` through the HOCON/JSON parse, `ReadonlyConfig`, and the shade round trip, not only in the test. If (b) holds, the `Jdbc.md` sentence is accurate and F5 can be closed together with F2.
If the truncated part also covered the rest of the four open points, please re-post that as well so I can mark them off. Thanks!
<!-- streview-comment:1346 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5628357382
@luozihen Thanks for the update — the pattern-caching commit is the right direction, and the bounded LRU shape (access-ordered `LinkedHashMap` capped at 256, wrapped in `Collections.synchronizedMap`) is fine as-is; the benign double-compile race is not a concern since `Pattern.compile()` is deterministic. Go ahead and push, and ping me once CI is green so I can take a final pass. If you have a rough sense of how many distinct patterns a busy multi-tenant deployment would configure, a one-line note in the field comment or PR description that 256 sits comfortably above that would be nice, but it's not a blocker.
On the rendering question: nothing looks truncated from where I'm reading, so I wouldn't spend more time chasing that — if a specific comment seems cut off on your end, just quote the part you're unsure about and I'll restate it.
To make the final pass quick, could you add a short per-item status for the earlier findings so I'm not guessing what the latest commit covers?
- **F1 (ReadonlyConfig.toConfig() dotted-key round-trip)** — is the API-level behavior change still in the PR, or has it been narrowed to the JDBC sink? This is the one I most want to confirm.
- **F2 / F5 (declaration-order semantics)** — with the cached patterns, how is "first pattern wins" now guaranteed given the option is a `Map<String, Object>`? Either an order-preserving structure or a doc change to match actual behavior works; please say which one you chose.
- **F3 (resolved key columns flowing into `PRIMARY_KEYS` / generated SQL)** — is there identifier validation on the resolved column names now?
- **F4 (`${primary_key}`/`${unique_key}` vs. engine-level TablePlaceholder)** — a sentence on which layer expands what, and in what order, would settle this.
- **F6 (regex validation → JDBC-12)** — I assume the cache compiles once; does an invalid pattern now fail fast at factory time with the JDBC-12 error code?
- **F7 (tests)** — is there a regression test for the old `toConfig()` dotted-key expansion, and an IT/E2E covering the new option?
- **F8 (`multi-table_config` naming)** — has the key been made consistent snake_case?
Just "done / not yet / intentionally unchanged because …" per item is plenty. Thanks again for sticking with it.
<!-- streview-comment:949 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5645072535
@SEZ9 @luozihen I independently re-pulled the head (`81f97896`) and checked F1 against both files directly rather than the status summary.
**`ReadonlyConfig#toConfig()`** — byte-for-byte identical to `dev` (base `1a8c637e996e`): `diff` between the two produces no output. So the API-level surface really is untouched, not just "functionally equivalent."
**`MultiTableFailureHelper.mergeOptions()`** — confirmed it no longer round-trips through HOCON:
```java
// dev:
return ReadonlyConfig.fromConfig(primary.toConfig().withFallback(fallback.toConfig()));
// this PR:
Map<String, Object> merged = new HashMap<>();
merged.putAll(fallback.getSourceMap());
merged.putAll(primary.getSourceMap());
return ReadonlyConfig.fromMap(merged);
```
On your ask — whether a key present in both sides resolves the same way as before — there's a real semantic difference worth naming precisely, not just "looks fine": the old `Config#withFallback` merges nested objects recursively (if both sides have an object at the same path, sub-fields from both survive, primary's sub-fields win on conflict), while the new code does a **shallow top-level merge** — if the same top-level key holds a nested map on both sides, `primary`'s whole nested value replaces `fallback`'s wholesale, and any of `fallback`'s sibling sub-fields not present in `primary` are silently dropped.
So this is a real narrowing of `mergeOptions()`'s contract, but I traced every current caller and none of them can actually hit the diverging case:
- `MultipleTableJobConfigParser.java:746`, and the Spark (`seatunnel-spark-2-starter`/`seatunnel-spark-starter-common`) and Flink (`seatunnel-flink-13-starter`/`seatunnel-flink-starter-common`) `SinkExecuteProcessor`s all call it as `mergeOptions(sinkConfig, envConfig)` — sink-plugin options and job-env options are disjoint namespaces; there's no key a JDBC/any sink config would set that env config also sets as a nested object.
- `withMultiTableFailurePolicy`/`withFailedTables` call it as `mergeOptions(singleKeyMap, options)`, where `singleKeyMap` only ever contains `MULTI_TABLE_FAILURE_POLICY.key()` or `MULTI_TABLE_INITIAL_FAILED_TABLES.key()` — neither collides with anything a sink config would set.
- `MultiTableFailureHelperTest#testMergeOptionsPreservesSpecialCharacterKeys` covers `multi_table_config` present only on the `primary` side plus a genuinely overlapping scalar key (`url`, primary wins) — it doesn't exercise the "same key nested on both sides" case, because no real caller produces that input today.
So F1 holds for every actual code path in the repo right now. The one thing I'd ask for, since this is a shared helper used by three engines (Zeta, Spark, Flink), not just this PR's JDBC feature: a one-line doc comment on `mergeOptions()` stating the merge is shallow/top-level (primary replaces fallback's value wholesale on key collision, not a recursive object merge) so a future caller doesn't assume HOCON-style deep merge like the old implementation had. Not a blocker on my side — just want that contract written down rather than implicit, given how many call sites depend on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5674347303
Thanks for pinning down exactly what's outstanding, @SEZ9 — and thanks @luozihen for posting F1-F8 as separate comments so nothing gets clipped.
Quick status check before answering: the head is still `2c5982be87e742c197143c64b8143ebff0fba2a5`, the same one my last full review (a documentation-only Javadoc commit on top of `81f97896`, `+9/-0` on `MultiTableFailureHelper.mergeOptions()`, verified by direct diff) already covered end to end. So there's no new code here for either of us to re-check — just closing the loop on the items you listed as open.
F1-F8 were independently re-verified against the actual source across two of my own rounds, not taken on trust from status summaries:
- **F1** — confirmed via direct diff: `ReadonlyConfig#toConfig()` is byte-for-byte identical to `dev`. The fix lives entirely in `MultiTableFailureHelper.mergeOptions()`, which I traced call-site by call-site (`MultipleTableJobConfigParser.java:746`, the Spark/Flink `SinkExecuteProcessor`s, `withMultiTableFailurePolicy`/`withFailedTables`) to confirm none of them can hit the shallow-vs-recursive-merge divergence this narrowing introduces.
- **F2/F5** — `toCompiledPatternMap` builds an explicit `LinkedHashMap<Pattern, List<String>>`, so "first declared pattern wins" is guaranteed by construction rather than incidental map order; `testResolveMultiTablePrimaryKeysFirstMatchWins` asserts it.
- **F3** — `validatePrimaryKeyColumns` is wired into the shared `applyPrimaryKeys` helper, so both the new multi-table-match branch and the legacy top-level `primary_keys` branch reject blank/comma-containing resolved columns before the join.
- **F4** — `TablePlaceholderProcessor.replaceTablePlaceholder` runs first but only rewrites top-level String/single-element-String-List values, so the nested `multi_table_config` map is untouched until the connector's own `expandPrimaryKeyPlaceholder` runs; a missing PK/unique-key fails fast with `JdbcConnectorException`/`JDBC-12` rather than leaving a literal placeholder or falling back silently.
- **F6** — `toCompiledPatternMap` compiles every declared pattern eagerly via `compilePattern` before any table matching, so an invalid regex fails fast with `JDBC-12` at factory time, not lazily at match time.
- **F8** — confirmed no stale `multi-table_config` spelling remains anywhere in code, error strings, tests, or EN/ZH docs.
That lines up, item for item, with what @luozihen just posted above — I'd treat those as accurate rather than needing separate re-verification of each pointer, since they match what I already found independently on this exact head.
From my side there's nothing open on the current head: Ready to merge, with only the one pre-existing Low/non-blocking carryover noted in my last review (`applyFallbackPrimaryKeys` duplicating the `getPrimaryKeyColumns`/`getUniqueKeyColumns` lookups instead of delegating to them). Happy to take another look if you find something on the actual diff that contradicts this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5691471135
@DanielLeens
Hi, could I ask for some advice? The latest CI keeps failing repeatedly in a set of Paimon tests. I’ve clicked “rerun fail job” several times, but it still doesn’t pass. As far as I can tell, since the last time CI was green, the only thing I submitted was a description.
I found the following error in the logs, which seems to point to a Testcontainers/Docker image pull issue rather than the test logic itself:
```
Status 404: {"message":"pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}
```
It occurs when Testcontainers tries to pull minio/minio:latest. I’m not sure whether this is a CI registry/auth issue, a Docker Hub rate-limit/permission issue, or something related to the image tag.
Would you have any suggestions on how to get CI green again? For example, should I check Docker login/credentials in CI, pin a specific MinIO image instead of latest, or configure a mirror/proxy? I may be missing something obvious, so any guidance would be greatly appreciated. If this is a known CI/infra issue, please let me know and I can wait or retry accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4311533879
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
thanks,
For example, what other configurations may he need to support? In the current situation, rapid failure has the lowest cost of understanding for users
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4311350861
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
I understand—this was taken into consideration in our earlier discussions.
The conclusion from our previous discussion was that MULTI_TABLE_CONFIG is designed to support future extension with other table-level configurations. The absence of primary_keys simply means that there is no primary-key-related customization. If we failed directly in that case, this parameter might not be able to support other configurations in the future. Therefore, I think the current behavior is reasonable: we should fail only when the primary_keys within MULTI_TABLE_CONFIG do not meet the configuration requirements, rather than failing directly just because primary_keys is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6310063380
@SEZ9 thanks for the follow-up. Your three open items (F6, F7, F8) are answered against this exact head, `82fc5e61a`, in my approving review posted at 00:41Z (the head has not changed since). Condensed, so you can verify directly:
- **F6 (invalid regex):** every pattern in the map is compiled up front in `JdbcSinkFactory.toCompiledPatternMap` (`JdbcSinkFactory.java:345`) via `compilePattern` (`:362`), which raises `INVALID_MULTI_TABLE_CONFIG` (the JDBC-12 family) with the offending pattern. So a bad regex fails when the sink is created, not lazily when a particular table is matched. The `optionRule` extension (`JdbcMultiTableConfigConditionExtension`) only validates the shape. Tests are `JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysInvalidRegexFails` and `#testCompilePatternInvalidFails`.
- **F7 (engine-level coverage):** `JdbcMysqlMultipleTablesIT#testMysqlJdbcMultipleTableE2e` now runs `jdbc_mysql_source_and_sink_with_multi_table_config.conf` (in `connector-jdbc-e2e-part-1`), so the option is exercised through the engine, not only the factory.
- **F8 (key naming):** the final user-facing key is `multi_table_config`; there is no `multi-table_config` left in the PR diff (code option and both docs use the snake_case spelling).
CI for this head: the apache-side `Build` check is `pass`, so I have no remaining non-CI blocker. A write-capable maintainer still needs to give the final approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6331300208
Thanks @luozihen, both fixes sound right.
1. Key column validation: having `toPrimaryKeyList()` reject non-string / null list elements and empty results with a JDBC-12 error that names the pattern, while keeping `validatePrimaryKeyColumns()` as the final guard, is the shape I was hoping for. Good that the new `JdbcSinkFactoryTest` cases cover both `resolveMultiTablePrimaryKeys()` and the `createSink` path.
2. REST JSON body: moving `ConfigBuilder.of(Map)` onto the same `rebuildWithLiteralKeys` helper as the config-file and shade paths closes the gap, and `testOfMapPreservesLiteralRegexOptionKeys` covering the map → `of(Map)` → `ConfigShadeUtils.decryptConfig` sequence is a solid regression test. My preference is to keep the REST commit in this PR so the literal-key behaviour is fixed in one place for all three entry points; the separate REST PR can be put on hold or closed once this lands.
Remaining asks before I can approve:
- Please post the CI result for d9bb1574f once it finishes.
- Could you confirm whether the earlier points are covered on this head or still pending: declaration-order guarantees for the pattern map (and the matching docs wording), the interaction with the engine-level `TablePlaceholder` replacement, a regression test for the old `toConfig()` dotted-key expansion plus an engine-level IT/E2E for the new option, and the `multi-table_config` key naming.
<!-- streview-comment:1932 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "hutiefang76 (via GitHub)" <gi...@apache.org>.
hutiefang76 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6332627570
@luozihen, there is real overlap at `ConfigBuilder.of(Map)`, variable backfill and the decrypt re-parse, but the two current implementations have different key semantics.
Your [JSON rebuild](https://github.com/apache/seatunnel/blob/d9bb1574f0dfee9f6673533dff2616522a76e6c4/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/ConfigBuilder.java#L283) keeps every map key literal. #12443 quotes only invalid path expressions and preserves the existing handling of valid ones. For example, `{"a->b":"v"}` still becomes nested `a.b` in #12443, whereas the JSON rebuild keeps one `a->b` key. Surrounding whitespace and quotes in valid keys differ too; ordinary dotted keys such as `job.mode` stay literal in both.
#12443 also covers [ReadonlyConfig.toConfig()](https://github.com/apache/seatunnel/blob/3d6c855f30d0a0b915befbe29abea3e31e6e7bb3/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java#L79), the config-adapter conversion and both variable-map conversions, which remain outside the three rebuild sites changed here. Its REST submit-path tests are separate from the JDBC mapping coverage.
I propose keeping one shared config fix in #12443 and rebasing the JDBC feature onto it, while keeping the primary-key validation changes in this PR. That avoids merging competing conversions. The intended key contract and merge order need to be agreed before either version lands; the two patches cannot simply be treated as identical fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6396812888
+1 to @SEZ9's request to settle the dotted-key contract with the REST-shaped test already proposed here. Preserving valid existing path semantics matters as much as accepting the regex key: a literal rebuild must not silently change how env's `shade.identifier` and `job.mode` are resolved.
Please push the map -> `RestUtil.buildConfig` -> `ConfigShadeUtils.decryptConfig` regression case with the valid dotted keys and the regex key together, and make the semantics table match its assertions. If literal reconstruction changes the valid-key behavior, the narrow invalid-path rescue is the safer direction than changing `toConfig()` globally. The head is unchanged, so I cannot treat that test or compatibility concern as fixed yet. I will wait for the code update; this is clarification of the existing points, not an additional review finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6421464248
@luozihen Thank you for your contribution. Moving forward with this work after 3.1.0🔥
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5595019491
@luozihen Thanks for the detailed status — here's my response point by point.
**F1 / F4 — direction looks right; closure pending verification.** Reverting `ReadonlyConfig#toConfig()` to `ConfigFactory.parseMap(confData)` and scoping the change to `MultiTableFailureHelper.mergeOptions()` is exactly what I was hoping for, and your explanation of why `TablePlaceholderProcessor.replaceTablePlaceholder` leaves the nested map untouched is consistent with the intent. I'll confirm both against the actual changes on the latest head before marking them resolved.
**F2 / F5 — yes, please make it stricter.** Relying on Jackson happening to produce a `LinkedHashMap` is an implementation detail rather than a contract, and since the docs promise "first pattern in declaration order wins", I'd like that pinned down in code:
- Make the ordering explicit where the option is read/iterated (e.g. copy into a `LinkedHashMap`, or type it so order is guaranteed by construction).
- Add a unit test declaring several patterns that match the same table and asserting the first-declared one wins.
**F3 — please handle in this PR.** Since the resolved key columns are comma-joined into `PRIMARY_KEYS` and end up in generated SQL, please validate each resolved column as a plain identifier (at minimum reject empty names) before joining, with a small unit test for a rejected value.
**F6 — please also do this in this PR.** Lazy failure at match time means a bad regex only surfaces once data flows for a matching table. Please compile every pattern during config validation and raise `JDBC-12` up front, with a unit test for an invalid pattern.
**F7 — two pieces:**
- The regression test for the restored `toConfig()` dotted-key expansion directly protects the F1 revert, so please add it in this PR.
- The engine-level IT/E2E for the new option can be a follow-up; please note that in the PR description.
**F8 — please rename.** The existing JDBC sink options are snake_case, so the mixed hyphen/underscore in `multi-table_config` will stand out to users. A fully snake_case name would fit the convention; please update the docs and tests to match. I understand the current name came from the issue discussion, but I'd rather keep the option surface consistent.
Summary for this PR: F2/F5 explicit ordering + test, F3 identifier validation + test, F6 eager regex validation + test, F7 `toConfig()` regression test, F8 rename. F1/F4 to be confirmed against the diff; F7's IT/E2E can be a follow-up. Ping me once pushed and I'll take another pass.
<!-- streview-comment:916 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5825482928
@luozihen thanks for mapping F1-F8 to concrete locations against head 00acc23a5 — that makes the re-review much easier.
I still need to walk the actual diff before marking anything resolved, so a few things that would help me close the items out:
- F1: please confirm in the PR description that `ReadonlyConfig.toConfig()` is left unchanged in this PR and briefly describe what the changes at `ConfigShadeUtils.java:224` and `MultiTableFailureHelper.java:64` do instead, so the compatibility impact on other callers is clear.
- F2/F5: a `LinkedHashMap` in `toCompiledPatternMap` preserves the order it receives, but "first pattern in declaration order wins" also depends on the parsed `Map<String, Object>` arriving in HOCON declaration order. Could you either add a unit test with two overlapping patterns declared in a known order that asserts the first one wins end to end from a parsed config, or soften the docs to say behaviour is undefined for overlapping patterns? Either is fine with me.
- F4: please add a sentence to the option docs stating whether `${primary_key}`/`${unique_key}` are expanded by the connector before or after the engine-level table placeholder replacement, so users know which one applies if both appear.
- F7: could you summarize what `ReadableConfigTest` and `JdbcMysqlMultipleTablesIT` assert for the dotted-key round-trip and the new option, respectively?
Once I have those and have gone through the diff for F3, F6 and F8, I'll post the updated status.
<!-- streview-comment:1289 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5807024685
@SEZ9
I have rewritten the "Review status (F1-F8)" section so each item points at the concrete code and test: F1 (ReadonlyConfig.java:76, ConfigShadeUtils.java:224, MultiTableFailureHelper.java:64), F2/F5 (JdbcSinkFactory.java:344/:314), F3 (:282), F4 (expandPrimaryKeyPlaceholder, :416), F6 (compilePattern, :361), F7 (ReadableConfigTest + JdbcMysqlMultipleTablesIT), F8 (JdbcSinkOptions.java:112). The line numbers are against the current head 00acc23a5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5861641995
@DanielLeens
Thank you for your patience in helping push this work forward and for patiently helping answer the other reviewer's question about being unable to retrieve the full comment—I really appreciate it. Regarding the CI status, two checks are still red. I have retried 8 times, but all-connectors-it-2 (8, ubuntu-latest)/all-connectors-it-2 (11, ubuntu-latest) still show as failed. I checked the run logs, and they seem to point again to a Docker image issue: Can't get Docker image: RemoteDockerImage(imageName=[quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z](https://quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z)
I would like to know whether this is an intermittent/flaky occurrence or an infrastructure change. Should I keep retrying, or take some other action to get CI green? Thank you very much for your guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi-table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5738485145
@luozihen Thanks for the detailed walkthrough on c15f9fd.
**F1** — Understood. If `ReadonlyConfig#toConfig()` is unchanged and `ReadonlyConfig.java` is not in this PR's diff, then the finding as written was aimed at the wrong place, and I'm happy to retarget it to `ConfigShadeUtils.processConfig`. The reasoning for the JSON round-trip there makes sense — a regex key like `^t_nova_.*$` being re-read as a HOCON path and failing with `ConfigException$BadPath` would block this feature. Two things I'd still like so we can close F1 and F7 together:
1. Confirm that existing configs using dotted keys still resolve to the same values after `processConfig` as before. Since the map holds already-parsed values I expect they do, but a small unit test asserting both cases (literal regex key preserved, dotted key unchanged) would make the guarantee explicit.
2. Your comment cuts off at "it merges the two option maps" — could you finish that thought on `MultiTableFailureHelper#mergeOptions()`? Specifically, how does the merge handle a key present in both maps, and does it keep the same precedence `withFallback()` gave?
**F4** — Good to see a test added for the placeholder pass ordering; that was the piece I most wanted covered.
**F2, F3, F5, F6, F8** — You mention these were pointed at the current source, but that part of the comment didn't come through. Could you re-post the pointers? In particular, for F2/F5 I'd like to see where declaration order for the map-typed option is preserved (or where the docs were adjusted to stop promising it), and for F6 where the user regex is compiled up front and mapped to the JDBC-12 error code.
Once those land I'll do a final pass against c15f9fd.
<!-- streview-comment:1148 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-5787712307
@luozihen Thanks for adding the clarifications to the "Review status (F1-F8)" section — the explanations for F1 (behaviour-neutral for other options and the `mergeOptions` callers), F2/F5 (HOCON string → `ReadonlyConfig` → option → `toCompiledPatternMap`), and F3 (dialect quoting wraps without escaping) all make sense to me.
Before I close these out, I'd like to tie each one to the code in `00acc23a582` rather than to the description alone. Could you point me to:
- **F1/F7** — the test that covers the old `toConfig()` dotted-key expansion, so the neutrality claim is backed by an assertion rather than prose.
- **F2/F5** — the specific test/assertion that exercises the ordering chain you described end to end.
- **F4/F6/F8** — the tests referenced in the description and the final option key name as it appears in the code.
- **F3** — a follow-up issue for the dialect-specific character rule, linked from the PR description so it doesn't get lost.
Once I can verify those in the diff, I have nothing further within this review scope. Thanks for the targeted updates.
<!-- streview-comment:1240 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6096314464
Hi @luozihen, thanks for pulling the logs and asking before retrying further — please stop retrying, it won't help here.
The `Can't get Docker image ... quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z` failure in `all-connectors-it-2 (8, ubuntu-latest)` / `all-connectors-it-2 (11, ubuntu-latest)` is deterministic, not flaky: that MinIO image is no longer pullable, so every run that reaches an S3/MinIO-based IT fails at container start regardless of your change. Nothing in this PR touches those tests. The fix is already on `dev` (the S3/MinIO ITs now use a digest-pinned public mirror), but your head `00acc23a5` is behind `dev` and still carries the old tag.
Concrete next steps:
1. Sync your branch with the latest `dev` (merge or rebase, whichever you prefer) and push.
2. Please do not hand-edit any of the MinIO test files in this PR — the sync brings in the correct versions.
3. Let CI run on the new head. If `all-connectors-it-2` still fails after the sync, paste the new failing job link here and I'll look at it against the updated head.
On the code side there is no new blocker from the earlier review; the remaining gate for this PR is a green CI run on the synced head.
<!-- streview-comment:1646 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6104499650
@luozihen Thanks for the update, and glad the MinIO problem went away after syncing with `dev`.
On the `elasticsearch-connector-it` failure: every commit on this branch after `00acc23a5` (`0361012392`, `1e667b60c`, `8a3d619c9`) is a merge of `dev`, and the diff of the PR-owned files between `00acc23a5` and `8a3d619c9` is empty, so the author code is unchanged since my last review and that failure is unlikely to come from this PR's own changes. Please don't add an ES-specific workaround to this branch; once `dev` is fixed, merge it again and re-run.
For the PR itself, I have no new source-side concerns on `8a3d619c9`. Since the author code is unchanged since `00acc23a5`, the points from my earlier review that are still open stand as written there. Thanks for your patience with the CI.
<!-- streview-comment:1659 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6143673372
@SEZ9 thanks for the item-by-item pass on `e761f0c5b`. I re-checked each point against the pinned head (author code is byte-identical to `00acc23a5`), and most of them are already covered in the diff. Details below so you can confirm against the files.
- **F1 (`ReadonlyConfig#toConfig()`):** agreed, it is not in the diff. The regex-key handling is in `ConfigShadeUtils.processConfig` and `MultiTableFailureHelper.mergeOptions`.
- **F2 / F5 (first declared pattern wins):** `toCompiledPatternMap` builds a `LinkedHashMap` (`JdbcSinkFactory.java:344`), and the contract is stated in the Javadoc and in `docs/en/connectors/sink/Jdbc.md` ("If a table matches multiple patterns, the first pattern in declaration order wins"). The end-to-end ordering is pinned by `JdbcSinkFactoryTest#testResolveMultiTablePrimaryKeysFromHoconFirstMatchWins` (`JdbcSinkFactoryTest.java:348`): it parses a real HOCON string with three overlapping patterns (`^table2$`, `^table1$`, `^table.*$`) and asserts that `table1` resolves to the `^table1$` mapping, not the later wildcard. `testResolveMultiTablePrimaryKeysFirstMatchWins` (line 329) covers the same rule with an explicit `LinkedHashMap`. So the "show where order is enforced plus a two-overlapping-patterns test" ask from option (a) is already met.
- **F3 (key columns flowing into SQL):** this one is only partly covered, and I want to be precise. `validatePrimaryKeyColumns` (`JdbcSinkFactory.java:282`) rejects blank names and names containing a comma, with `JDBC-12`. It does not check the column against the upstream schema, and I would not call that a blocker: the generated statements quote identifiers through the dialect (see the Javadoc at lines 270-280), and a column that does not exist surfaces as a clear DDL/DML error at the sink instead of silently writing wrong data. A schema-existence check would be a reasonable hardening, but it would also be a behaviour choice for the author and maintainers, so I would treat it as optional.
- **F4 (placeholder ordering, no-PK case):** `docs/en` and `docs/zh` `sink-options-placeholders.md` now describe the order (engine-level pass first, connector-level expansion per matched table at sink creation). The no-key case is an error, not an empty expansion: `expandPrimaryKeyPlaceholder` (`JdbcSinkFactory.java:416-443`) throws `JDBC-12` naming the table and pattern, documented in `Jdbc.md` and tested by `testResolveMultiTablePrimaryKeysMissingPrimaryKeyFails` (line 317). The pass order is tested by `testPlaceholderOrderBetweenEngineAndConnectorPasses` (line 503).
- **F6 (invalid regex):** `compilePattern` (`JdbcSinkFactory.java:361`) wraps `PatternSyntaxException` into `JDBC-12`, and it runs from `resolveMultiTablePrimaryKeys` during `createSink`, so the user sees the connector error code, not a raw regex exception. Tests: `testResolveMultiTablePrimaryKeysInvalidRegexFails` (line 399) and `testCompilePatternInvalidFails` (line 419). One small gap: these assert the exception type rather than the `JDBC-12` code. That is a Low polish item, not a blocker.
- **F7 (IT):** `JdbcMysqlMultipleTablesIT` runs a real MySQL-to-JDBC multi-table job with `jdbc_mysql_source_and_sink_with_multi_table_config.conf` and asserts the primary key columns that end up in the created sink tables (`mtc_table1` gets the composite key, `mtc_table2` a single key). The config also declares a catch-all `^mtc_.*$` last, so the job fails if declaration order were not preserved: this pins F2 end to end on a real job. To be precise about the remaining gap, both tables in the IT match a pattern, so the "one matched and one unmatched table" combination is covered only at unit level (`testResolveMultiTablePrimaryKeysUnmatched`, `JdbcSinkFactoryTest.java:307`, and the legacy-fallback factory tests), not in the IT. Adding a third source table that matches no pattern and asserting it keeps the fallback `id` key would close that, and I would treat it as a recommended, non-blocking addition.
- **F8 (naming):** the key is `multi_table_config` (`JdbcSinkOptions.java:113`) with the nested `primary_keys`, spelled the same way in code, docs and tests. A search of the head for the old `multi-table_config` form finds nothing.
**CI on `e761f0c5b` (fork run `38199352694`, completed):** 81 jobs passed, 11 skipped, 2 failed: `all-connectors-it-2 (11)` and `(17)`. Both fail on the same two tests, `CassandraIT#testCassandra` (line 134) and `#testCassandraMultiTableSource` (line 144, `expected: <0> but was: <1>`, container index 4). This PR touches no Cassandra files (0 of its 15 files), and every JDBC IT job, including the one with `JdbcMysqlMultipleTablesIT`, is green. I could not tell from the logs whether the Cassandra failure also reproduces on `dev`, so I am not claiming it is a known flake; the diff simply cannot reach it. The branch is 8 commits behind `dev` (`diverged`, ahead 26, behind 8).
Suggested next step for @luozihen: re-run only the failed jobs from the fork Actions page ("Re-run failed jobs"). Syncing `dev` is optional, and please do not change Cassandra tests in this PR.
My source-side conclusion is unchanged: no blocker; the optional items are the F3 schema check, an unmatched table in the IT (F7), and asserting the `JDBC-12` code in the invalid-regex tests. Thanks again for the careful follow-through!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6258694329
Thanks for the update, and thanks to davidzollo for the +1.
Before I sign off on head `231cbdea588a499c88d970ee6c1e031984e48f9b`, I'd like to tie the earlier review points back to the actual changes, since I haven't verified them against the diff myself:
- **F1 (`ReadonlyConfig.toConfig()` dotted-key semantics):** could you point me to the change that keeps the `toConfig()` behavior unchanged for existing callers, and to the regression test that locks the dotted-key lookup?
- **F2 / F5 (first pattern in declaration order wins):** could you point to where the declaration order is preserved for the pattern map, and to the test that parses real HOCON with non-alphabetical patterns and asserts the first match wins?
- **F8 (option key naming):** which spelling is final for the user-facing option key — `multi-table_config` or `multi_table_config` — and is it the same in the option definition and the docs?
The remaining Medium items (F3 identifier validation of the comma-joined primary keys, F4 interaction with the engine-level placeholder replacement, F6 explicit regex validation mapped to `JDBC-12`, F7 engine-level IT/E2E coverage) don't need to block this focused change; a follow-up issue tracking them would be appreciated so they don't get lost.
Once the above is confirmed, I'm happy to approve. Nice work on this.
<!-- streview-comment:1862 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "luozihen (via GitHub)" <gi...@apache.org>.
luozihen commented on code in PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#discussion_r4311597198
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkFactory.java:
##########
@@ -192,6 +178,310 @@ public TableSink createSink(TableSinkFactoryContext context) {
finalCatalogTable);
}
+ /**
+ * Writes the resolved primary key columns into the sink config map and rebuilds the catalog
+ * table so that auto-created tables and generated upsert/update/delete statements use the given
+ * key columns.
+ *
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable applyPrimaryKeys(
+ Map<String, String> map, CatalogTable catalogTable, List<String> primaryKeys) {
+ validatePrimaryKeyColumns(primaryKeys, catalogTable.getTablePath().getTableName());
+ map.put(JdbcSinkOptions.PRIMARY_KEYS.key(), String.join(",", primaryKeys));
+ return buildCatalogTableWithPrimaryKey(catalogTable, primaryKeys);
+ }
+
+ /**
+ * Rebuilds the catalog table so that its primary key is the given columns. The option map is
+ * left untouched, so a caller that resolved the columns from the existing configuration keeps
+ * the value the user wrote.
+ *
+ * @param catalogTable the table being processed
+ * @param primaryKeys the resolved key columns
+ * @return a new catalog table whose primary key is replaced with the resolved columns
+ */
+ private CatalogTable buildCatalogTableWithPrimaryKey(
+ CatalogTable catalogTable, List<String> primaryKeys) {
+ PrimaryKey configPk =
+ PrimaryKey.of(
+ catalogTable.getTablePath().getTableName() + "_config_pk", primaryKeys);
+ TableSchema tableSchema = catalogTable.getTableSchema();
+ return CatalogTable.of(
+ catalogTable.getTableId(),
+ TableSchema.builder()
+ .primaryKey(configPk)
+ .constraintKey(tableSchema.getConstraintKeys())
+ .columns(tableSchema.getColumns())
+ .build(),
+ catalogTable.getOptions(),
+ catalogTable.getPartitionKeys(),
+ catalogTable.getComment(),
+ catalogTable.getCatalogName());
+ }
+
+ /**
+ * Resolves the primary key columns using the pre-existing fallback logic when no multi-table
+ * mapping matches: explicit top-level {@code primary_keys}, otherwise the catalog primary key,
+ * otherwise the first unique key. When no key can be determined, the config map is left
+ * unchanged and the sink falls back to plain INSERT.
+ *
+ * @param config the sink config
+ * @param map the sink config map that is later turned back into a {@link ReadonlyConfig}
+ * @param catalogTable the table being processed
+ * @return the (possibly rebuilt) catalog table
+ */
+ private CatalogTable applyFallbackPrimaryKeys(
+ ReadonlyConfig config, Map<String, String> map, CatalogTable catalogTable) {
+ PrimaryKey primaryKey = catalogTable.getTableSchema().getPrimaryKey();
+ if (CollectionUtils.isEmpty(config.get(JdbcSinkOptions.PRIMARY_KEYS))) {
+ if (primaryKey != null && !CollectionUtils.isEmpty(primaryKey.getColumnNames())) {
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ String.join(",", primaryKey.getColumnNames()));
+ } else {
+ Optional<ConstraintKey> keyOptional =
+ catalogTable.getTableSchema().getConstraintKeys().stream()
+ .filter(
+ key ->
+ ConstraintKey.ConstraintType.UNIQUE_KEY.equals(
+ key.getConstraintType()))
+ .findFirst();
+ keyOptional.ifPresent(
+ constraintKey ->
+ map.put(
+ JdbcSinkOptions.PRIMARY_KEYS.key(),
+ constraintKey.getColumnNames().stream()
+ .map(
+ ConstraintKey.ConstraintKeyColumn
+ ::getColumnName)
+ .collect(Collectors.joining(","))));
+ }
+ return catalogTable;
+ }
+ // The legacy explicit primary_keys branch keeps the pre-existing behaviour: only the
+ // catalog primary key is rebuilt, the option value is neither rewritten nor re-validated.
+ return buildCatalogTableWithPrimaryKey(
+ catalogTable, config.get(JdbcSinkOptions.PRIMARY_KEYS));
+ }
+
+ /**
+ * Validates that each resolved primary key column is a non-empty plain identifier that does not
+ * contain a comma, so it can be safely comma-joined into {@code PRIMARY_KEYS} and used in
+ * generated SQL. Generated statements quote identifiers through the dialect, so a blank name
+ * and a comma (the delimiter of the comma-joined {@code PRIMARY_KEYS} value) are the only
+ * inputs that cannot be represented safely.
+ *
+ * @param primaryKeys the resolved key columns
+ * @param tableName the table being processed, used in the error message
+ * @throws JdbcConnectorException when a column name is blank or contains a comma
+ */
+ private void validatePrimaryKeyColumns(List<String> primaryKeys, String tableName) {
+ for (String key : primaryKeys) {
+ if (StringUtils.isBlank(key)) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column for table '%s' is empty.", tableName));
+ }
+ if (key.contains(",")) {
+ throw new JdbcConnectorException(
+ JdbcConnectorErrorCode.INVALID_MULTI_TABLE_CONFIG,
+ String.format(
+ "Resolved primary key column '%s' for table '%s' must not contain a comma.",
+ key, tableName));
+ }
+ }
+ }
+
+ /**
+ * Resolves the per-table primary key mapping from {@code multi_table_config.primary_keys}.
+ *
+ * <p>Each key is a Java regular expression matched against the upstream table name using full
+ * match semantics; the first matching pattern in declaration order wins. Every pattern is
+ * compiled eagerly so an invalid expression fails fast with {@code JDBC-12} instead of
+ * surfacing lazily once a matching table is processed. An unmatched table returns {@link
+ * Optional#empty()}, leaving the fallback logic to run.
+ *
+ * @param config the sink config
+ * @param catalogTable the upstream table; its name is matched against the patterns and its
+ * schema is used to expand {@code ${primary_key}} / {@code ${unique_key}}
+ * @return the resolved key columns, or {@link Optional#empty()} when no pattern matches
+ */
+ Optional<List<String>> resolveMultiTablePrimaryKeys(
+ ReadonlyConfig config, CatalogTable catalogTable) {
+ Map<String, Object> multiTableConfig = config.get(JdbcSinkOptions.MULTI_TABLE_CONFIG);
+ if (multiTableConfig == null || multiTableConfig.isEmpty()) {
+ return Optional.empty();
+ }
+ Object primaryKeysObj = multiTableConfig.get("primary_keys");
Review Comment:
For example, with configurations such as data_save_mode / schema_save_mode, I can’t rule out scenarios where different configurations may be needed for different tables. Of course, this is just a habitual precaution on my part. As for the current situation, failing fast is completely fine; I can accept either failing fast or keeping the current implementation. Thank you for your review and suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2] Support per-table primary key mapping via multi_table_config.primary_keys for JDBC sink [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12015:
URL: https://github.com/apache/seatunnel/pull/12015#issuecomment-6414884620
Hi @luozihen, thanks for the clear plan, and for telling everyone before holding the branch.
This sounds right to me, and it matches the direction we converged on:
- #12443 ("Preserve literal config keys across REST and file loading") is still open, on head `509fddab`; I approved it separately. Landing the config-key conversion there first, so this PR does not carry a competing one, is the safest order.
- After #12443 merges, syncing `dev` and dropping `0c43f10dd` (the `ConfigBuilder.of(Map)` commit) removes the compatibility concern I raised on the REST path. The sync should also clear the unrelated `LLMBooleanOutputTest` failure in the current Build, which is already fixed on `dev` (#12938).
- Please keep the REST-shaped regression (`map -> RestUtil.buildConfig -> ConfigShadeUtils.decryptConfig`) and the `toConfig()` assertions, rebased on top of #12443's contract, with `shade.identifier`, `job.mode`, `a.b`, `a->b` and a regex key asserted together, so the test pins the final behavior. If the Jdbc docs (en and zh) still claim that regex keys work for the REST JSON body, please keep that wording consistent with whatever #12443 actually delivers.
- The JDBC `multi_table_config.primary_keys` feature and its validation stay as they are.
The head is still `d9bb1574`, so my existing review stands for now and I cannot mark the REST point as resolved yet. Once you post the rebased head, I will do a full re-review of it and update my review state accordingly. No action is needed from you until #12443 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org