You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JNSimba (via GitHub)" <gi...@apache.org> on 2026/05/28 09:33:12 UTC
[PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
JNSimba opened a new pull request, #63833:
URL: https://github.com/apache/doris/pull/63833
## Summary
- Default-skip flink-cdc's in-snapshot backfill on the from-to path so large splits no longer accumulate the entire chunk + backfill stream in the fetcher's outputBuffer; from-to is at-least-once and tolerates the duplicates this introduces. TVF (job-driven and standalone) keeps the standard `false` default for exactly-once via per-task offset commit.
- Expose `skip_snapshot_backfill` as a user-facing property with strict `true`/`false` validation on both from-to (CREATE JOB) and TVF (SELECT FROM cdc_stream(...)) entry points.
- Fix snapshot completion under `pollWithoutBuffer`: a split is now marked complete only after its high-watermark event has been consumed (`splitState.getHighWatermark() != null`), not on the first non-empty fetcher batch. Without this, enabling the new default truncates any split larger than debezium's `max.batch.size` and yields an NPE on offset extraction.
- Read `streaming_task_timeout_multiplier` live in `StreamingMultiTblTask.isTimeout()` so `admin set frontend config` affects already-running tasks, matching the `@ConfField(mutable=true)` contract.
## Test plan
- [ ] \`mvn compile\` passes for \`fe-core\` and \`cdc_client\`
- [ ] New \`test_streaming_postgres_job_snapshot_fat_split\` / \`test_streaming_mysql_job_snapshot_fat_split\` pass: 2100 rows with \`snapshot_split_size=3000\` (single split exceeds \`max.batch.size=2048\`), asserting count=2100, distinct=2100, \`id BETWEEN 2049 AND 2100\`=52, and post-snapshot DML still flows
- [ ] Existing \`test_streaming_*_id_gap_completeness\` / \`test_streaming_*_snapshot\` / \`test_streaming_*_async_split*\` regressions still pass
- [ ] Validator rejects \`skip_snapshot_backfill=foo\` at SQL analysis on both CREATE JOB and \`cdc_stream\` TVF
- [ ] \`admin set frontend config ("streaming_task_timeout_multiplier"="N")\` while a from-to task is running takes effect on the running task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4564552823
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/13)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63833_c99f868bfac95c74ae612246262d8e48c7491646/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63833_c99f868bfac95c74ae612246262d8e48c7491646/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4563857785
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4562715893
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4562630395
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4564142243
<details>
<summary>TPC-DS: <b>Total hot run time: 170787 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c99f868bfac95c74ae612246262d8e48c7491646, data reload: false
query5 4308 664 533 533
query6 329 217 208 208
query7 4266 548 327 327
query8 333 235 221 221
query9 8813 4015 3994 3994
query10 477 355 292 292
query11 5762 2421 2194 2194
query12 188 130 125 125
query13 1285 608 402 402
query14 6114 5422 5132 5132
query14_1 4421 4429 4420 4420
query15 211 203 182 182
query16 1019 449 424 424
query17 1144 722 587 587
query18 2515 481 359 359
query19 218 203 157 157
query20 134 135 131 131
query21 216 132 117 117
query22 13662 13661 13394 13394
query23 17411 16566 16225 16225
query23_1 16381 16309 16388 16309
query24 7647 1755 1308 1308
query24_1 1324 1320 1321 1320
query25 600 491 438 438
query26 1338 330 179 179
query27 2656 600 356 356
query28 4492 2012 2029 2012
query29 1020 640 521 521
query30 302 240 203 203
query31 1126 1078 959 959
query32 89 77 77 77
query33 547 367 309 309
query34 1198 1155 624 624
query35 793 817 722 722
query36 1390 1414 1265 1265
query37 151 109 99 99
query38 3255 3177 3088 3088
query39 946 918 912 912
query39_1 873 875 877 875
query40 238 152 128 128
query41 70 68 67 67
query42 113 111 110 110
query43 332 331 290 290
query44
query45 223 207 200 200
query46 1133 1169 723 723
query47 2384 2395 2253 2253
query48 409 425 314 314
query49 654 502 413 413
query50 945 357 257 257
query51 4393 4289 4360 4289
query52 105 104 97 97
query53 255 287 203 203
query54 337 282 278 278
query55 99 95 88 88
query56 329 323 315 315
query57 1459 1429 1314 1314
query58 307 276 275 275
query59 1642 1656 1462 1462
query60 316 317 302 302
query61 160 154 156 154
query62 702 653 582 582
query63 243 200 207 200
query64 2429 813 635 635
query65
query66 1729 476 357 357
query67 29867 29854 29546 29546
query68
query69 462 341 311 311
query70 1041 1004 1015 1004
query71 315 274 265 265
query72 3006 2717 2228 2228
query73 877 810 428 428
query74 5132 4984 4800 4800
query75 2721 2577 2277 2277
query76 2290 1127 751 751
query77 413 404 340 340
query78 12531 12444 11879 11879
query79 1480 1041 747 747
query80 643 539 448 448
query81 457 289 247 247
query82 1346 158 120 120
query83 356 277 249 249
query84 275 140 110 110
query85 903 537 467 467
query86 389 356 334 334
query87 3459 3377 3233 3233
query88 3593 2741 2687 2687
query89 450 396 352 352
query90 1944 181 176 176
query91 178 168 138 138
query92 82 82 75 75
query93 1448 1424 917 917
query94 536 375 306 306
query95 686 378 345 345
query96 1012 761 367 367
query97 2764 2708 2629 2629
query98 239 227 225 225
query99 1167 1147 1023 1023
Total cold run time: 254947 ms
Total hot run time: 170787 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4570261480
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4563567074
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/13)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63833_41fd120977678cad7af8f166add762bbfe7c0d92/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63833_41fd120977678cad7af8f166add762bbfe7c0d92/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4563257223
<details>
<summary>TPC-DS: <b>Total hot run time: 171582 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41fd120977678cad7af8f166add762bbfe7c0d92, data reload: false
query5 4301 663 538 538
query6 335 216 201 201
query7 4282 589 309 309
query8 334 239 225 225
query9 8769 4066 4098 4066
query10 450 344 298 298
query11 5843 2529 2207 2207
query12 174 129 127 127
query13 1287 607 456 456
query14 6110 5466 5151 5151
query14_1 4492 4496 4466 4466
query15 218 204 190 190
query16 1052 464 456 456
query17 1179 766 625 625
query18 2775 492 366 366
query19 233 210 171 171
query20 145 143 134 134
query21 219 140 119 119
query22 13675 13569 13320 13320
query23 17461 16569 16227 16227
query23_1 16400 16274 16450 16274
query24 7531 1767 1298 1298
query24_1 1299 1291 1332 1291
query25 540 459 421 421
query26 1315 311 175 175
query27 2685 572 356 356
query28 4405 1980 2031 1980
query29 966 628 496 496
query30 314 235 196 196
query31 1133 1070 950 950
query32 92 75 71 71
query33 531 357 304 304
query34 1196 1147 677 677
query35 767 783 704 704
query36 1427 1410 1222 1222
query37 156 102 93 93
query38 3221 3147 3136 3136
query39 929 919 894 894
query39_1 887 904 880 880
query40 233 141 121 121
query41 65 64 63 63
query42 108 111 107 107
query43 335 339 290 290
query44
query45 214 206 202 202
query46 1103 1232 764 764
query47 2380 2358 2315 2315
query48 411 460 288 288
query49 629 489 384 384
query50 1052 350 253 253
query51 4349 4287 4289 4287
query52 109 121 96 96
query53 256 287 197 197
query54 316 269 265 265
query55 94 90 84 84
query56 304 303 292 292
query57 1437 1430 1309 1309
query58 310 278 269 269
query59 1602 1698 1444 1444
query60 319 325 305 305
query61 154 154 148 148
query62 710 649 590 590
query63 250 211 202 202
query64 2375 805 648 648
query65
query66 1689 488 367 367
query67 29740 29575 29509 29509
query68
query69 512 348 302 302
query70 1024 974 1003 974
query71 311 280 269 269
query72 3020 2708 2403 2403
query73 891 762 431 431
query74 5150 4962 4785 4785
query75 2698 2596 2253 2253
query76 2310 1173 807 807
query77 397 414 341 341
query78 12396 12507 11850 11850
query79 1273 1094 735 735
query80 578 544 452 452
query81 454 276 233 233
query82 238 154 119 119
query83 267 277 242 242
query84 263 147 109 109
query85 887 647 526 526
query86 359 338 338 338
query87 3407 3373 3237 3237
query88 3642 2757 2764 2757
query89 436 405 356 356
query90 2195 184 188 184
query91 198 185 159 159
query92 84 81 79 79
query93 1499 1448 904 904
query94 561 373 340 340
query95 696 431 448 431
query96 1050 865 343 343
query97 2739 2740 2612 2612
query98 237 233 228 228
query99 1187 1157 1031 1031
Total cold run time: 253414 ms
Total hot run time: 171582 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4563839695
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4563177605
<details>
<summary>TPC-H: <b>Total hot run time: 31642 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41fd120977678cad7af8f166add762bbfe7c0d92, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17651 4032 3999 3999
q2 q3 10759 1364 792 792
q4 4686 479 350 350
q5 7586 2256 2095 2095
q6 237 174 136 136
q7 938 785 643 643
q8 9369 1741 1606 1606
q9 5174 4866 4872 4866
q10 6392 2200 1885 1885
q11 440 265 242 242
q12 630 430 299 299
q13 18118 3405 2744 2744
q14 270 257 237 237
q15 q16 825 777 708 708
q17 1003 965 934 934
q18 6886 5602 5692 5602
q19 1306 1292 1111 1111
q20 505 496 314 314
q21 6438 2806 2699 2699
q22 457 380 386 380
Total cold run time: 99670 ms
Total hot run time: 31642 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4833 4681 4686 4681
q2 q3 5024 5365 4681 4681
q4 2087 2180 1380 1380
q5 4960 4658 4666 4658
q6 232 177 138 138
q7 1933 1757 1572 1572
q8 2409 2094 2087 2087
q9 7830 7326 7331 7326
q10 4738 4656 4193 4193
q11 523 397 359 359
q12 721 726 526 526
q13 3005 3441 2792 2792
q14 269 289 251 251
q15 q16 675 694 599 599
q17 1279 1250 1250 1250
q18 7279 6978 6799 6799
q19 1120 1100 1125 1100
q20 2203 2213 1925 1925
q21 5217 4567 4335 4335
q22 519 468 441 441
Total cold run time: 56856 ms
Total hot run time: 51093 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4565185019
# FE Regression Coverage Report
Increment line coverage ` 4.44% (2/45)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63833_c99f868bfac95c74ae612246262d8e48c7491646_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63833_c99f868bfac95c74ae612246262d8e48c7491646_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63833:
URL: https://github.com/apache/doris/pull/63833#discussion_r3317856339
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java:
##########
@@ -276,6 +276,8 @@ private void checkRequiredSourceProperties() {
if (!sourceProperties.containsKey(DataSourceConfigKeys.OFFSET)) {
sourceProperties.put(DataSourceConfigKeys.OFFSET, DataSourceConfigKeys.OFFSET_LATEST);
}
+ // from-to is at-least-once; default-skip in-snapshot backfill.
+ sourceProperties.putIfAbsent(DataSourceConfigKeys.SKIP_SNAPSHOT_BACKFILL, "true");
Review Comment:
Defaulting from-to jobs to `skip_snapshot_backfill=true` can lose source changes that commit while a snapshot split is being read. With this flag, `IncrementalSourceScanFetcher.pollWithoutBuffer()` returns the snapshot stream without the buffered backfill reconciliation, and the existing snapshot-to-binlog handoff still stores each finished split's high watermark (`JdbcSourceOffsetProvider.updateOffset`) and creates the binlog split from the minimum finished high watermark (`MySqlSourceReader.createBinlogSplit`, with the same pattern for JDBC/Postgres). For a single split, an update/delete/insert between that split's low and high watermark is neither emitted by the snapshot backfill nor replayed by the subsequent binlog split starting at the high watermark. Please either keep the default as `false`, or change the handoff so skipped-backfill snapshots resume binlog from the corresponding low watermark/earlier offset and rely on idempotent replays. The existing concurrent-DML snapsh
ot tests also need to run under the new default to cover this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4562654353
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63833:
URL: https://github.com/apache/doris/pull/63833#issuecomment-4564061870
<details>
<summary>TPC-H: <b>Total hot run time: 31508 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c99f868bfac95c74ae612246262d8e48c7491646, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17705 4161 4028 4028
q2 q3 10789 1372 811 811
q4 4686 478 340 340
q5 7556 2255 2079 2079
q6 238 175 133 133
q7 914 782 635 635
q8 9519 1739 1527 1527
q9 5162 4948 4953 4948
q10 6438 2202 1896 1896
q11 436 276 251 251
q12 632 416 291 291
q13 18148 3363 2831 2831
q14 261 257 249 249
q15 q16 829 763 711 711
q17 999 866 955 866
q18 6907 5645 5506 5506
q19 1310 1215 1118 1118
q20 559 541 280 280
q21 6311 2849 2699 2699
q22 451 370 309 309
Total cold run time: 99850 ms
Total hot run time: 31508 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5044 4784 4676 4676
q2 q3 4968 5264 4605 4605
q4 2149 2201 1365 1365
q5 4812 4860 4677 4677
q6 233 175 126 126
q7 1853 1818 1564 1564
q8 2395 2159 2174 2159
q9 7964 7844 7398 7398
q10 4726 4647 4210 4210
q11 542 381 348 348
q12 736 733 512 512
q13 3008 3385 2775 2775
q14 269 274 256 256
q15 q16 669 693 628 628
q17 1266 1241 1234 1234
q18 7364 6958 6751 6751
q19 1135 1060 1089 1060
q20 2224 2220 1954 1954
q21 5240 4569 4423 4423
q22 530 445 420 420
Total cold run time: 57127 ms
Total hot run time: 51141 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](streaming-job) avoid potential OOM when reading large snapshot splits [doris]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #63833:
URL: https://github.com/apache/doris/pull/63833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org