You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JNSimba (via GitHub)" <gi...@apache.org> on 2026/05/21 11:00:34 UTC
[PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
JNSimba opened a new pull request, #63490:
URL: https://github.com/apache/doris/pull/63490
## Summary
- Add an optional `server_id` source property for MySQL CDC streaming jobs. Accepts a single value (e.g. `5400`) or a range (e.g. `5400-5408`). When unset, the value is derived from the jobId hash so existing jobs keep their current server_id when `snapshot_parallelism = 1`.
- Fix a latent collision: when `snapshot_parallelism > 1` and source-side DML happens during snapshot, all parallel `SnapshotSplitReader` instances previously shared the same server_id and their backfill BinaryLogClient connections kicked each other out of MySQL's dump-thread slot, dropping binlog events between low and high watermark. Each subtask now gets a distinct server_id from the resolved range; the single binlog reader uses the range start.
- Cross-field check: reject `server_id` range width smaller than `snapshot_parallelism` at job startup with a clear fix-it suggestion.
## Test plan
- [x] `ConfigUtilTest`: 15 cases covering default-derive (Integer.MAX_VALUE hash clamp, hash=0 bump), user single value / range, malformed input, blank input, range-vs-parallelism width check, non-positive parallelism.
- [ ] `test_streaming_mysql_job_server_id.groovy`: 4 rejection cases (format / zero / backward / width) under `offset=latest` for synchronous CREATE JOB feedback, plus 3 happy-path cases verifying snapshot data syncs under single value / range / default-derive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4561489226
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4514621252
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4514620968
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4532758038
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4543629254
run vault_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4543921957
# FE Regression Coverage Report
Increment line coverage ` 3.10% (4/129)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63490_a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63490_a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4542981067
<details>
<summary>TPC-H: <b>Total hot run time: 31842 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17645 4084 4092 4084
q2 q3 10787 1399 821 821
q4 4681 475 350 350
q5 7554 2250 2118 2118
q6 255 180 141 141
q7 917 774 637 637
q8 9395 1786 1640 1640
q9 6963 5027 5002 5002
q10 6425 2223 1879 1879
q11 434 273 244 244
q12 696 421 295 295
q13 18245 3460 2787 2787
q14 270 252 238 238
q15 q16 825 778 704 704
q17 1010 1008 930 930
q18 6822 5790 5644 5644
q19 1218 1420 1139 1139
q20 513 418 281 281
q21 5977 2756 2584 2584
q22 449 373 324 324
Total cold run time: 101081 ms
Total hot run time: 31842 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4945 4731 5001 4731
q2 q3 4967 5269 4631 4631
q4 2138 2220 1432 1432
q5 4792 4852 4670 4670
q6 233 176 135 135
q7 1829 1726 1607 1607
q8 2429 1982 1936 1936
q9 7421 7507 7560 7507
q10 4782 4713 4226 4226
q11 550 438 367 367
q12 741 744 542 542
q13 3071 3415 2868 2868
q14 271 291 252 252
q15 q16 675 710 604 604
q17 1302 1266 1265 1265
q18 7552 6960 6754 6754
q19 1130 1095 1078 1078
q20 2245 2234 1963 1963
q21 5293 4657 4517 4517
q22 529 460 414 414
Total cold run time: 56895 ms
Total hot run time: 51499 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4507513780
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4507515435
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4507506586
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4543421994
# FE UT Coverage Report
Increment line coverage ` 71.11% (32/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63490_a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63490_a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4509089122
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63490_5696cc20023456eb0d5eddbf100a80cc2c60e531_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63490_5696cc20023456eb0d5eddbf100a80cc2c60e531_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4508148796
<details>
<summary>TPC-DS: <b>Total hot run time: 169540 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5696cc20023456eb0d5eddbf100a80cc2c60e531, data reload: false
query5 4316 669 528 528
query6 355 222 201 201
query7 4280 570 303 303
query8 331 239 226 226
query9 8826 4045 4045 4045
query10 468 373 301 301
query11 5825 2410 2223 2223
query12 172 129 127 127
query13 1290 612 450 450
query14 5996 5377 5073 5073
query14_1 4523 4394 4369 4369
query15 209 204 187 187
query16 1029 467 433 433
query17 1162 735 606 606
query18 2741 512 360 360
query19 223 209 182 182
query20 142 131 133 131
query21 219 138 117 117
query22 13624 13662 13447 13447
query23 17291 16382 15967 15967
query23_1 16069 16193 16179 16179
query24 7458 1747 1297 1297
query24_1 1343 1292 1296 1292
query25 595 489 438 438
query26 1303 325 181 181
query27 2687 571 339 339
query28 4435 1977 1980 1977
query29 1016 643 516 516
query30 308 242 203 203
query31 1119 1072 946 946
query32 97 78 81 78
query33 552 363 314 314
query34 1176 1101 650 650
query35 774 823 671 671
query36 1292 1290 1209 1209
query37 149 103 89 89
query38 3216 3147 3069 3069
query39 937 917 904 904
query39_1 865 872 877 872
query40 233 148 122 122
query41 67 65 63 63
query42 109 112 109 109
query43 323 327 285 285
query44
query45 214 200 193 193
query46 1068 1199 701 701
query47 2284 2315 2112 2112
query48 405 408 304 304
query49 628 498 381 381
query50 941 335 263 263
query51 4319 4323 4240 4240
query52 107 106 94 94
query53 250 282 203 203
query54 315 270 259 259
query55 95 90 86 86
query56 308 327 304 304
query57 1392 1376 1293 1293
query58 307 278 275 275
query59 1544 1641 1399 1399
query60 334 323 315 315
query61 159 156 158 156
query62 679 629 549 549
query63 244 198 206 198
query64 2405 794 626 626
query65
query66 1662 482 352 352
query67 29681 29973 29965 29965
query68
query69 469 340 295 295
query70 1034 942 984 942
query71 310 278 276 276
query72 3060 2733 2369 2369
query73 856 802 429 429
query74 5050 4992 4709 4709
query75 2677 2627 2259 2259
query76 2297 1129 751 751
query77 395 401 328 328
query78 12232 12277 11589 11589
query79 1475 1026 760 760
query80 914 533 448 448
query81 503 276 236 236
query82 1364 159 127 127
query83 358 292 265 265
query84 256 145 109 109
query85 938 539 462 462
query86 452 355 303 303
query87 3473 3361 3243 3243
query88 3529 2687 2682 2682
query89 442 381 334 334
query90 1780 187 189 187
query91 183 169 142 142
query92 81 77 72 72
query93 1471 1450 835 835
query94 624 354 305 305
query95 682 471 358 358
query96 1014 824 368 368
query97 2693 2723 2585 2585
query98 237 243 250 243
query99 1095 1104 1022 1022
Total cold run time: 253400 ms
Total hot run time: 169540 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63490:
URL: https://github.com/apache/doris/pull/63490#discussion_r3286821445
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java:
##########
@@ -118,6 +119,45 @@ public static void validateSource(Map<String, String> input,
"ssl_mode '" + DataSourceConfigKeys.SSL_MODE_VERIFY_CA
+ "' requires ssl_rootcert to be set");
}
+
+ validateServerIdConfig(input);
+ }
+
+ // Shared by validateSource and the cdc_stream TVF entrypoint so both reject malformed
+ // server_id at SQL-analysis time, not as a cdc_client runtime error.
+ public static void validateServerIdConfig(Map<String, String> input)
+ throws IllegalArgumentException {
+ String serverIdValue = input.get(DataSourceConfigKeys.SERVER_ID);
+ if (serverIdValue == null) {
+ return;
+ }
+ int[] range = parseServerIdRange(serverIdValue);
+ if (range == null) {
+ throw new IllegalArgumentException(
+ "Invalid value for key '" + DataSourceConfigKeys.SERVER_ID + "': "
+ + serverIdValue
+ + ". Expected a single value (e.g. '5400') or range (e.g. '5400-5408')"
+ + " with start >= 1 and start <= end.");
+ }
+ String parallelismValue = input.getOrDefault(
+ DataSourceConfigKeys.SNAPSHOT_PARALLELISM,
Review Comment:
This cross-field check is correct for CREATE, but the ALTER JOB path currently calls `validateSource(this.getSourceProperties(), ...)` with only the delta map before `StreamingInsertJob.alterJob()` merges it into the existing source properties. That means an existing MySQL job created with `"snapshot_parallelism" = "4"` can be altered with only `"server_id" = "99001"`; this code defaults the missing parallelism to 1 and accepts it, but the effective runtime config after merge is width 1 with parallelism 4, so later snapshot readers will hit the same invalid configuration this PR is trying to reject at analysis time. Please validate the merged source properties in the ALTER path, or reject changing `server_id` for existing jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4539811391
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4540312403
# FE UT Coverage Report
Increment line coverage ` 74.47% (35/47)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63490_918be4b864583885cb43290315b60963efa4f209/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63490_918be4b864583885cb43290315b60963efa4f209/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4541279661
<details>
<summary>TPC-DS: <b>Total hot run time: 172440 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a68a33eeca17b81c6b13ff89018f6f396b2d57c8, data reload: false
query5 4327 698 528 528
query6 341 217 207 207
query7 4237 568 316 316
query8 337 234 219 219
query9 8830 4114 4109 4109
query10 456 340 300 300
query11 5765 2615 2261 2261
query12 179 128 125 125
query13 1283 643 438 438
query14 6149 5539 5254 5254
query14_1 4560 4593 4566 4566
query15 216 213 185 185
query16 1055 487 418 418
query17 1174 749 623 623
query18 2504 497 371 371
query19 218 208 169 169
query20 139 135 134 134
query21 216 143 121 121
query22 13761 13606 13476 13476
query23 17410 16674 16356 16356
query23_1 16490 16559 16439 16439
query24 7542 1799 1320 1320
query24_1 1337 1321 1341 1321
query25 592 516 448 448
query26 1309 327 180 180
query27 2689 577 345 345
query28 4448 2043 2054 2043
query29 1002 652 530 530
query30 297 239 207 207
query31 1148 1092 965 965
query32 92 77 77 77
query33 564 361 329 329
query34 1219 1092 655 655
query35 768 789 708 708
query36 1379 1443 1252 1252
query37 158 109 91 91
query38 3236 3200 3060 3060
query39 921 926 908 908
query39_1 900 881 863 863
query40 239 145 124 124
query41 64 65 63 63
query42 113 105 111 105
query43 335 338 300 300
query44
query45 214 204 194 194
query46 1089 1197 721 721
query47 2369 2364 2211 2211
query48 421 393 315 315
query49 621 489 388 388
query50 958 364 271 271
query51 4468 4345 4217 4217
query52 105 107 98 98
query53 253 280 205 205
query54 311 275 262 262
query55 95 90 86 86
query56 312 302 302 302
query57 1459 1399 1343 1343
query58 295 270 270 270
query59 1609 1691 1466 1466
query60 326 338 322 322
query61 162 155 152 152
query62 702 652 596 596
query63 248 202 207 202
query64 2398 822 616 616
query65
query66 1716 487 371 371
query67 29906 29948 29035 29035
query68
query69 456 343 304 304
query70 1054 994 1007 994
query71 295 273 275 273
query72 3012 2741 2422 2422
query73 850 737 420 420
query74 5157 5010 4830 4830
query75 2687 2620 2287 2287
query76 2302 1159 779 779
query77 398 446 339 339
query78 12367 12549 11877 11877
query79 1475 1022 791 791
query80 644 531 445 445
query81 453 284 247 247
query82 1359 158 121 121
query83 364 293 258 258
query84 267 148 106 106
query85 908 544 448 448
query86 407 353 352 352
query87 3461 3387 3222 3222
query88 3626 2785 2744 2744
query89 448 391 347 347
query90 1921 187 180 180
query91 180 167 143 143
query92 84 88 73 73
query93 1646 1477 939 939
query94 557 357 304 304
query95 682 479 349 349
query96 1044 754 357 357
query97 2748 2755 2625 2625
query98 234 232 233 232
query99 1186 1156 1030 1030
Total cold run time: 255326 ms
Total hot run time: 172440 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4543080739
<details>
<summary>TPC-DS: <b>Total hot run time: 170745 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a31e7d7a0d4b6bab4467798ffb1011d73bdc4fb1, data reload: false
query5 4354 651 514 514
query6 336 225 202 202
query7 4226 536 320 320
query8 322 232 217 217
query9 8821 4080 4070 4070
query10 455 352 296 296
query11 5816 2574 2238 2238
query12 178 127 126 126
query13 1285 612 426 426
query14 6189 5517 5178 5178
query14_1 4469 4496 4471 4471
query15 209 209 188 188
query16 1030 449 422 422
query17 1122 718 597 597
query18 2521 466 353 353
query19 205 194 164 164
query20 133 130 130 130
query21 215 140 119 119
query22 13666 13602 13379 13379
query23 17374 16580 16181 16181
query23_1 16448 16410 16400 16400
query24 7498 1733 1318 1318
query24_1 1339 1327 1358 1327
query25 580 513 463 463
query26 1311 342 183 183
query27 2682 569 357 357
query28 4455 1982 1973 1973
query29 1046 664 517 517
query30 313 240 212 212
query31 1134 1077 958 958
query32 90 79 77 77
query33 561 385 311 311
query34 1172 1124 659 659
query35 763 816 706 706
query36 1445 1417 1268 1268
query37 156 106 88 88
query38 3242 3124 3081 3081
query39 928 922 912 912
query39_1 875 884 882 882
query40 253 148 134 134
query41 78 67 67 67
query42 114 111 109 109
query43 331 337 294 294
query44
query45 220 211 216 211
query46 1091 1200 756 756
query47 2437 2365 2228 2228
query48 400 414 304 304
query49 633 499 381 381
query50 994 350 247 247
query51 4369 4317 4238 4238
query52 109 107 93 93
query53 248 283 194 194
query54 316 275 247 247
query55 95 90 85 85
query56 309 291 309 291
query57 1449 1425 1362 1362
query58 303 277 271 271
query59 1588 1681 1442 1442
query60 321 325 297 297
query61 163 153 161 153
query62 695 654 591 591
query63 244 200 200 200
query64 2373 798 630 630
query65
query66 1707 491 361 361
query67 29716 29043 29527 29043
query68
query69 460 346 304 304
query70 1007 1024 961 961
query71 293 273 268 268
query72 3118 2962 2482 2482
query73 873 783 408 408
query74 5118 5008 4780 4780
query75 2684 2631 2272 2272
query76 2330 1170 775 775
query77 414 407 333 333
query78 12451 12450 11950 11950
query79 1494 1011 732 732
query80 652 543 463 463
query81 455 283 245 245
query82 1318 157 120 120
query83 359 273 242 242
query84 276 144 116 116
query85 893 552 456 456
query86 400 369 361 361
query87 3432 3381 3251 3251
query88 3585 2731 2721 2721
query89 436 403 346 346
query90 1914 181 183 181
query91 173 167 144 144
query92 78 82 71 71
query93 1519 1467 900 900
query94 552 365 277 277
query95 684 402 352 352
query96 1085 805 331 331
query97 2747 2747 2625 2625
query98 249 229 232 229
query99 1153 1152 1033 1033
Total cold run time: 254618 ms
Total hot run time: 170745 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4542517397
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4543206392
run valut_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4551403868
run vault_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4516557862
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63490:
URL: https://github.com/apache/doris/pull/63490#discussion_r3280689193
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java:
##########
@@ -53,7 +53,8 @@ public class DataSourceConfigValidator {
DataSourceConfigKeys.SSL_MODE,
DataSourceConfigKeys.SSL_ROOTCERT,
DataSourceConfigKeys.SLOT_NAME,
- DataSourceConfigKeys.PUBLICATION_NAME
+ DataSourceConfigKeys.PUBLICATION_NAME,
+ DataSourceConfigKeys.SERVER_ID
);
Review Comment:
Adding `server_id` to the allowed key set without validating its syntax/range lets invalid persisted jobs through FE validation. The reader-side check in `ConfigUtil.resolveServerIdRange()` is only reached when MySQL source config is built; `/api/initReader` only calls `Env.getReader()` -> `MySqlSourceReader.initialize()`, which initializes the serializer/thread pool and does not parse `server_id`, and initial/snapshot jobs skip reader initialization at create time entirely. For example, `CREATE JOB ... FROM MYSQL (... "offset" = "latest", "server_id" = "abc")` passes this validator and `/api/initReader`, then fails later when a task calls `getSourceSplits()`/`prepareBinlogSplit()`. That also means the new regression cases expecting synchronous CREATE failure for malformed `server_id` will not catch the error at the point they assert. Please validate `server_id` here for MySQL, including single/range format, `>= 1`, start <= end, and width vs `snapshot_parallelism` (and reject or
ignore it explicitly for non-MySQL sources).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4508056734
<details>
<summary>TPC-H: <b>Total hot run time: 30804 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5696cc20023456eb0d5eddbf100a80cc2c60e531, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17640 3859 3869 3859
q2 q3 10793 1436 822 822
q4 4696 490 356 356
q5 7667 2272 2112 2112
q6 373 178 138 138
q7 1006 763 643 643
q8 9364 1701 1473 1473
q9 6653 4961 4905 4905
q10 6440 2136 1808 1808
q11 441 277 248 248
q12 695 422 296 296
q13 18307 3456 2821 2821
q14 261 252 240 240
q15 q16 818 762 701 701
q17 917 1002 969 969
q18 6876 5642 5422 5422
q19 1161 1294 1030 1030
q20 511 407 256 256
q21 5694 2661 2402 2402
q22 434 354 303 303
Total cold run time: 100747 ms
Total hot run time: 30804 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4165 4144 4082 4082
q2 q3 4510 4900 4365 4365
q4 2119 2236 1410 1410
q5 4391 4309 4278 4278
q6 229 170 129 129
q7 2181 1920 1666 1666
q8 2525 2133 2062 2062
q9 7801 7802 7815 7802
q10 4507 4457 4100 4100
q11 569 465 383 383
q12 730 737 588 588
q13 3404 3671 3091 3091
q14 296 340 292 292
q15 q16 732 746 703 703
q17 1356 1305 1310 1305
q18 8229 7458 7051 7051
q19 1134 1087 1133 1087
q20 2217 2219 1934 1934
q21 5352 4870 4521 4521
q22 530 495 417 417
Total cold run time: 56977 ms
Total hot run time: 51266 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4540858507
<details>
<summary>TPC-H: <b>Total hot run time: 31215 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 918be4b864583885cb43290315b60963efa4f209, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17717 4105 4078 4078
q2 q3 10856 1384 829 829
q4 4681 476 353 353
q5 7637 2247 2134 2134
q6 239 170 137 137
q7 982 760 635 635
q8 9352 1724 1666 1666
q9 5140 4979 4954 4954
q10 6363 2221 1871 1871
q11 452 279 243 243
q12 635 430 288 288
q13 18116 3403 2732 2732
q14 276 255 237 237
q15 q16 828 779 707 707
q17 936 884 961 884
q18 7211 5897 5550 5550
q19 1312 1367 1053 1053
q20 539 399 257 257
q21 5849 2634 2304 2304
q22 438 359 303 303
Total cold run time: 99559 ms
Total hot run time: 31215 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4407 4391 4283 4283
q2 q3 4544 5016 4397 4397
q4 2125 2242 1410 1410
q5 4497 4351 4365 4351
q6 321 289 164 164
q7 2287 1915 1720 1720
q8 2554 2230 2227 2227
q9 8391 7966 8060 7966
q10 4868 4775 4361 4361
q11 597 417 406 406
q12 780 757 547 547
q13 3380 3649 3027 3027
q14 295 313 269 269
q15 q16 713 751 658 658
q17 1428 1314 1491 1314
q18 7848 7464 7349 7349
q19 1160 1154 1089 1089
q20 2238 2243 1974 1974
q21 5324 4697 4548 4548
q22 528 459 426 426
Total cold run time: 58285 ms
Total hot run time: 52486 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4540867713
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63490:
URL: https://github.com/apache/doris/pull/63490#discussion_r3303029847
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java:
##########
@@ -122,6 +123,45 @@ public static void validateSslVerifyCaPair(Map<String, String> input) throws Ill
"ssl_mode '" + DataSourceConfigKeys.SSL_MODE_VERIFY_CA
+ "' requires ssl_rootcert to be set");
}
+
+ validateServerIdConfig(input);
+ }
+
+ // Shared by validateSource and the cdc_stream TVF entrypoint so both reject malformed
+ // server_id at SQL-analysis time, not as a cdc_client runtime error.
+ public static void validateServerIdConfig(Map<String, String> input)
+ throws IllegalArgumentException {
+ String serverIdValue = input.get(DataSourceConfigKeys.SERVER_ID);
+ if (serverIdValue == null) {
+ return;
+ }
+ int[] range = parseServerIdRange(serverIdValue);
+ if (range == null) {
+ throw new IllegalArgumentException(
+ "Invalid value for key '" + DataSourceConfigKeys.SERVER_ID + "': "
+ + serverIdValue
+ + ". Expected a single value (e.g. '5400') or range (e.g. '5400-5408')"
+ + " with start >= 1 and start <= end.");
+ }
+ String parallelismValue = input.getOrDefault(
+ DataSourceConfigKeys.SNAPSHOT_PARALLELISM,
+ DataSourceConfigKeys.SNAPSHOT_PARALLELISM_DEFAULT);
+ Integer parallelism = parsePositiveInt(parallelismValue);
+ if (parallelism == null) {
+ throw new IllegalArgumentException(
+ "Invalid value for key '" + DataSourceConfigKeys.SNAPSHOT_PARALLELISM
+ + "': " + parallelismValue + ". Expected a positive integer.");
+ }
+ int width = range[1] - range[0] + 1;
+ // Range must cover every parallel SnapshotSplitReader; cdc_client throws otherwise.
Review Comment:
This width check is too strict for binlog-only startup modes. For `offset=latest`, `earliest`, a timestamp, or a specific binlog JSON offset, `MySqlSourceReader.getSourceSplits()` returns only `BINLOG_SPLIT_ID` and `getBinlogSplitReader()` creates config for subtask `0`, so a single configured `server_id` is sufficient even if `snapshot_parallelism` is larger. With the current validation, a valid job such as `offset=latest`, `snapshot_parallelism=4`, `server_id=5400` is rejected at CREATE time; the runtime resolver has the same unconditional width check as well. Please gate the range-width requirement to `initial`/`snapshot` startup modes, or otherwise pass the required reader count instead of always using `snapshot_parallelism`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #63490:
URL: https://github.com/apache/doris/pull/63490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4508195477
# FE UT Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63490_5696cc20023456eb0d5eddbf100a80cc2c60e531/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63490_5696cc20023456eb0d5eddbf100a80cc2c60e531/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4540967375
<details>
<summary>TPC-DS: <b>Total hot run time: 172998 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 918be4b864583885cb43290315b60963efa4f209, data reload: false
query5 4320 672 520 520
query6 337 224 202 202
query7 4245 571 305 305
query8 325 245 226 226
query9 8852 4197 4174 4174
query10 460 339 303 303
query11 5847 2396 2205 2205
query12 184 133 128 128
query13 1262 616 451 451
query14 6154 5556 5260 5260
query14_1 4508 4496 4480 4480
query15 213 205 184 184
query16 1023 448 356 356
query17 1134 737 584 584
query18 2553 485 345 345
query19 215 221 162 162
query20 144 135 129 129
query21 223 135 126 126
query22 13624 13927 13480 13480
query23 17427 16501 16417 16417
query23_1 16432 16472 16377 16377
query24 7506 1773 1321 1321
query24_1 1323 1365 1304 1304
query25 559 475 421 421
query26 1323 338 177 177
query27 2656 563 344 344
query28 4459 2050 2010 2010
query29 1021 637 531 531
query30 312 235 203 203
query31 1159 1099 961 961
query32 89 80 86 80
query33 564 368 307 307
query34 1221 1164 644 644
query35 787 816 694 694
query36 1424 1426 1307 1307
query37 154 114 98 98
query38 3256 3158 3107 3107
query39 934 931 903 903
query39_1 893 887 884 884
query40 245 149 134 134
query41 72 70 68 68
query42 111 111 113 111
query43 339 337 311 311
query44
query45 218 209 203 203
query46 1087 1259 740 740
query47 2377 2325 2258 2258
query48 420 418 320 320
query49 659 517 405 405
query50 1047 365 265 265
query51 4415 4319 4182 4182
query52 108 115 97 97
query53 255 280 213 213
query54 335 302 274 274
query55 103 95 93 93
query56 312 326 323 323
query57 1429 1441 1343 1343
query58 302 289 276 276
query59 1568 1693 1461 1461
query60 340 346 316 316
query61 189 182 185 182
query62 705 719 574 574
query63 251 203 211 203
query64 2409 813 653 653
query65
query66 1717 479 374 374
query67 29786 29723 29438 29438
query68
query69 463 356 312 312
query70 1021 989 1018 989
query71 320 278 258 258
query72 3166 2728 2468 2468
query73 847 765 446 446
query74 5110 4980 4861 4861
query75 2684 2645 2273 2273
query76 2295 1150 790 790
query77 403 419 345 345
query78 12356 12542 11916 11916
query79 1453 1052 710 710
query80 651 554 465 465
query81 451 281 247 247
query82 1369 170 128 128
query83 355 276 251 251
query84 265 144 113 113
query85 881 543 461 461
query86 419 357 304 304
query87 3450 3397 3258 3258
query88 3692 2769 2749 2749
query89 458 388 345 345
query90 1887 190 185 185
query91 182 170 144 144
query92 84 79 77 77
query93 1517 1369 865 865
query94 541 368 318 318
query95 702 462 356 356
query96 1088 835 358 358
query97 2765 2741 2620 2620
query98 233 233 247 233
query99 1187 1163 1041 1041
Total cold run time: 255651 ms
Total hot run time: 172998 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4541193567
<details>
<summary>TPC-H: <b>Total hot run time: 31835 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a68a33eeca17b81c6b13ff89018f6f396b2d57c8, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17648 4086 4026 4026
q2 q3 10823 1395 804 804
q4 4682 480 346 346
q5 7578 2290 2117 2117
q6 231 175 135 135
q7 911 809 650 650
q8 9426 1599 1610 1599
q9 5166 5016 4976 4976
q10 6370 2197 1880 1880
q11 445 281 243 243
q12 643 423 302 302
q13 18154 3478 2809 2809
q14 276 258 241 241
q15 q16 834 779 710 710
q17 999 966 1000 966
q18 7254 5802 5601 5601
q19 1179 1229 1151 1151
q20 613 451 311 311
q21 5907 2921 2651 2651
q22 599 384 317 317
Total cold run time: 99738 ms
Total hot run time: 31835 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4943 4950 4735 4735
q2 q3 4917 5294 4716 4716
q4 2284 2447 1509 1509
q5 5019 4793 4646 4646
q6 227 174 134 134
q7 1958 1767 1601 1601
q8 2506 2179 2191 2179
q9 8061 7483 7367 7367
q10 4791 4734 4265 4265
q11 548 425 407 407
q12 740 751 549 549
q13 2977 3345 2813 2813
q14 278 277 264 264
q15 q16 687 717 625 625
q17 1309 1278 1284 1278
q18 7579 6986 6942 6942
q19 1105 1097 1112 1097
q20 2245 2245 2006 2006
q21 5422 4738 4709 4709
q22 538 482 425 425
Total cold run time: 58134 ms
Total hot run time: 52267 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) support user-specified mysql server_id with per-reader assignment [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63490:
URL: https://github.com/apache/doris/pull/63490#issuecomment-4542516077
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org