You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JNSimba (via GitHub)" <gi...@apache.org> on 2026/05/19 09:13:55 UTC
[PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
JNSimba opened a new pull request, #63402:
URL: https://github.com/apache/doris/pull/63402
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
`JdbcPostgreSQLClient.getJdbcColumnsInfo` calls `DatabaseMetaData.getColumns(catalog, schemaPattern, tableNamePattern, columnNamePattern)`. Per the JDBC spec the 3rd argument is a **SQL LIKE pattern**, so literal `_` / `%` characters in the requested table name are interpreted as wildcards by the Postgres driver. When a streaming job is created with `include_tables = "user_info_pg_normal1"` and a neighbour table like `userXinfo_pg_normal1` happens to coexist in the same schema, the metadata query returns columns from **both** tables. The combined result then trips `CREATE TABLE` on the Doris side with errors such as `errCode = 2, detailMessage = Duplicate column name 'name'`, or pollutes the auto-created table schema with stray columns.
The repro is trivial: in the same Postgres schema create
- `user_info_pg_normal1(name varchar, age int2)` — the table we want to capture
- `userXinfo_pg_normal1(name varchar, weight float8)` — a decoy whose name only differs from the target by a single character that `_` matches
then run `CREATE JOB ... include_tables = "user_info_pg_normal1"`. Without the fix the schema fetched for the target leaks `weight` (or `Duplicate column name 'name'`, depending on column order).
Fix: after fetching the `ResultSet`, drop rows whose `TABLE_NAME` does not exactly equal the requested `remoteTableName`. This mirrors [apache/flink-cdc#4239 (FLINK-38965)](https://github.com/apache/flink-cdc/pull/4239), which solved the same class of issue in the Postgres CDC source. We deliberately do **not** escape `_` / `%` at the source: relying on `DatabaseMetaData.getSearchStringEscape()` is driver-version dependent (older Oracle drivers don't honour escape sequences in `getTables`), while filtering on the consumer side is deterministic and driver-agnostic.
Scope:
- Only `JdbcPostgreSQLClient` is patched. This is the path used by Postgres streaming jobs (the failing case). MySQL streaming jobs were checked against the same decoy pattern and do not reproduce the bug because MySQL Connector/J doesn't pull neighbour rows here in practice — so `JdbcMySQLClient` is left untouched in this PR.
- The JDBC catalog path lives in a separate module (`fe-connector-jdbc/.../JdbcConnectorClient`) and is **not** part of this PR. It already does partial escape but intentionally skips `_` / `%` for driver-compatibility reasons; a follow-up can apply the same after-the-fact filter there.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
A decoy table `userXinfo_pg_normal1` (with a different column shape: `weight float8`) is added to `test_streaming_postgres_job.groovy`, plus an `assert !createTalInfo.contains(\`weight\`)` guard. Without the fix the case fails (either `Duplicate column name` during `CREATE TABLE`, or the `weight` assert trips). The same decoy is mirrored into `test_streaming_mysql_job.groovy` as a baseline so any future regression in MySQL Connector/J's behaviour is caught immediately.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4542585469
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4541620769
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4543101763
<details>
<summary>TPC-DS: <b>Total hot run time: 172798 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9107947bb0fd67973c72d4792cd91618a3b2a1f, data reload: false
query5 4317 675 518 518
query6 335 232 200 200
query7 4211 546 313 313
query8 330 242 222 222
query9 8849 4126 4129 4126
query10 460 350 306 306
query11 5841 2600 2230 2230
query12 187 130 127 127
query13 1332 612 465 465
query14 6170 5578 5256 5256
query14_1 4578 4571 4555 4555
query15 217 207 184 184
query16 983 452 460 452
query17 1068 757 622 622
query18 2456 503 376 376
query19 218 211 167 167
query20 142 144 133 133
query21 219 143 117 117
query22 13713 13719 13376 13376
query23 17430 16557 16268 16268
query23_1 16462 16398 16432 16398
query24 7459 1796 1324 1324
query24_1 1339 1327 1344 1327
query25 592 527 442 442
query26 1309 317 180 180
query27 2692 571 361 361
query28 4441 1979 2001 1979
query29 1051 668 522 522
query30 313 253 202 202
query31 1135 1092 989 989
query32 95 82 78 78
query33 567 395 309 309
query34 1176 1134 658 658
query35 820 809 689 689
query36 1420 1364 1245 1245
query37 162 117 88 88
query38 3239 3180 3113 3113
query39 936 940 909 909
query39_1 886 864 881 864
query40 244 153 131 131
query41 71 70 70 70
query42 120 113 118 113
query43 351 346 312 312
query44
query45 218 212 205 205
query46 1119 1199 756 756
query47 2364 2426 2351 2351
query48 406 409 302 302
query49 640 504 383 383
query50 1004 354 261 261
query51 4345 4304 4231 4231
query52 103 105 95 95
query53 261 288 206 206
query54 316 274 257 257
query55 93 92 90 90
query56 310 309 292 292
query57 1442 1432 1357 1357
query58 305 275 270 270
query59 1597 1665 1473 1473
query60 334 327 310 310
query61 164 147 159 147
query62 692 649 599 599
query63 240 205 213 205
query64 2397 816 628 628
query65
query66 1732 486 360 360
query67 29920 29845 29755 29755
query68
query69 463 339 306 306
query70 1016 1079 997 997
query71 304 273 267 267
query72 3032 2660 2405 2405
query73 822 755 418 418
query74 5145 5031 4844 4844
query75 2707 2636 2319 2319
query76 2294 1147 781 781
query77 411 417 347 347
query78 12508 12392 11876 11876
query79 1523 1029 770 770
query80 1236 547 461 461
query81 506 292 243 243
query82 1449 155 125 125
query83 383 284 247 247
query84 287 142 112 112
query85 946 545 454 454
query86 435 335 334 334
query87 3496 3440 3247 3247
query88 3616 2735 2730 2730
query89 454 392 348 348
query90 1777 186 184 184
query91 179 167 137 137
query92 80 81 72 72
query93 1438 1516 883 883
query94 662 343 303 303
query95 697 399 356 356
query96 1125 775 339 339
query97 2724 2747 2575 2575
query98 245 229 231 229
query99 1184 1146 1052 1052
Total cold run time: 256058 ms
Total hot run time: 172798 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4541611350
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4541610932
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4486231700
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4542586581
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4541619897
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63402:
URL: https://github.com/apache/doris/pull/63402#discussion_r3302048315
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java:
##########
@@ -63,6 +63,10 @@ public List<JdbcFieldSchema> getJdbcColumnsInfo(String remoteDbName, String remo
String catalogName = getCatalogName(conn);
rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName);
while (rs.next()) {
+ // getColumns treats the table name as a LIKE pattern; drop rows pulled in via `_`/`%`.
+ if (!remoteTableName.equals(rs.getString("TABLE_NAME"))) {
Review Comment:
This still allows the same class of leak when the schema name contains JDBC LIKE wildcards. `getColumns(catalog, schemaPattern, tableNamePattern, ...)` treats both `schemaPattern` and `tableNamePattern` as patterns; this PR filters `TABLE_NAME`, but if the configured Postgres schema is `cdc_test` and another schema such as `cdcXtest` contains a table with the same name, those rows pass this check and their columns are mixed into `tableSchema`. Please also filter `TABLE_SCHEM` (or otherwise constrain the schema exactly) and add a regression with a decoy schema to cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4543966495
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/87)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63402_b9107947bb0fd67973c72d4792cd91618a3b2a1f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63402_b9107947bb0fd67973c72d4792cd91618a3b2a1f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63402:
URL: https://github.com/apache/doris/pull/63402#issuecomment-4543002879
<details>
<summary>TPC-H: <b>Total hot run time: 31614 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9107947bb0fd67973c72d4792cd91618a3b2a1f, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17685 4067 4010 4010
q2 q3 10773 1376 851 851
q4 4690 475 354 354
q5 7627 2271 2096 2096
q6 261 176 144 144
q7 942 821 635 635
q8 9352 1780 1650 1650
q9 6500 4955 4881 4881
q10 6458 2252 1858 1858
q11 440 270 242 242
q12 688 430 295 295
q13 18212 3366 2787 2787
q14 265 254 235 235
q15 q16 816 779 700 700
q17 883 894 875 875
q18 6947 5785 5556 5556
q19 1182 1292 1212 1212
q20 535 454 271 271
q21 5989 2782 2638 2638
q22 458 386 324 324
Total cold run time: 100703 ms
Total hot run time: 31614 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4776 4753 4976 4753
q2 q3 4901 5274 4665 4665
q4 2155 2218 1440 1440
q5 4928 4746 4746 4746
q6 240 181 124 124
q7 1799 1978 1579 1579
q8 2453 1978 1961 1961
q9 7404 7504 7375 7375
q10 4807 4699 4253 4253
q11 543 391 358 358
q12 738 753 547 547
q13 2962 3351 2789 2789
q14 274 278 253 253
q15 q16 684 707 614 614
q17 1302 1277 1263 1263
q18 7347 6734 6847 6734
q19 1151 1109 1109 1109
q20 2229 2219 1969 1969
q21 5343 4656 4528 4528
q22 521 482 405 405
Total cold run time: 56557 ms
Total hot run time: 51465 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63402:
URL: https://github.com/apache/doris/pull/63402#discussion_r3303144953
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java:
##########
@@ -63,6 +63,11 @@ public List<JdbcFieldSchema> getJdbcColumnsInfo(String remoteDbName, String remo
String catalogName = getCatalogName(conn);
rs = getRemoteColumns(databaseMetaData, catalogName, remoteDbName, remoteTableName);
while (rs.next()) {
+ // getColumns treats schema/table as LIKE patterns; drop rows pulled in via `_`/`%`.
Review Comment:
This fixes the legacy FE PostgreSQL client, but the same metadata path exists in the parallel connector implementation and is still vulnerable. `fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java` is adapted from this class; its `getJdbcColumnsInfo()` calls `getRemoteColumns(meta, cat, remoteDbName, remoteTableName)` and then unconditionally adds every returned `JdbcFieldInfo`. Since PostgreSQL JDBC treats both schema and table as LIKE patterns, a connector catalog over schema `cdc_test` can still mix columns from `cdcXtest.<same_table>` or from `userXinfo_pg_normal1`. Please mirror the exact `TABLE_SCHEM`/`TABLE_NAME` filtering there as well, and add connector coverage or otherwise justify why that path cannot be used for PostgreSQL catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #63402:
URL: https://github.com/apache/doris/pull/63402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org