You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2025/11/21 06:29:51 UTC
[PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
suxiaogang223 opened a new pull request, #56117:
URL: https://github.com/apache/doris/pull/56117
### What problem does this PR solve?
This pull request introduces a mechanism to limit the maximum number of file splits when querying external tables, which helps control resource usage and prevent excessive parallelism.
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3300903893
<details>
<summary>ClickBench: <b>Total hot run time: 30.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 006aefed780078304a28501c31e8a98605c031b3, data reload: false
query1 0.05 0.05 0.05
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.63 0.12 0.12
query5 0.27 0.28 0.26
query6 1.19 0.66 0.64
query7 0.03 0.03 0.03
query8 0.05 0.04 0.04
query9 0.61 0.52 0.53
query10 0.58 0.57 0.58
query11 0.17 0.12 0.11
query12 0.15 0.13 0.12
query13 0.63 0.63 0.62
query14 1.03 1.00 1.06
query15 0.87 0.84 0.86
query16 0.41 0.44 0.41
query17 1.07 1.02 1.05
query18 0.22 0.19 0.21
query19 1.99 1.90 1.86
query20 0.01 0.01 0.02
query21 15.43 0.95 0.58
query22 0.77 1.13 0.86
query23 14.76 1.41 0.62
query24 7.44 2.02 0.87
query25 0.44 0.22 0.08
query26 0.57 0.17 0.14
query27 0.07 0.06 0.05
query28 9.31 0.87 0.45
query29 12.55 3.98 3.27
query30 0.28 0.14 0.12
query31 2.83 0.61 0.39
query32 3.24 0.59 0.48
query33 3.17 3.11 3.17
query34 16.21 5.50 4.78
query35 4.91 4.86 4.90
query36 0.68 0.52 0.50
query37 0.11 0.07 0.07
query38 0.07 0.05 0.05
query39 0.03 0.04 0.03
query40 0.19 0.14 0.15
query41 0.09 0.04 0.02
query42 0.03 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 104.53 s
Total hot run time: 30.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3300895965
<details>
<summary>TPC-DS: <b>Total hot run time: 188762 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 006aefed780078304a28501c31e8a98605c031b3, data reload: false
query1 1103 439 410 410
query2 6592 1703 1685 1685
query3 6751 219 222 219
query4 26267 23662 23548 23548
query5 4401 653 485 485
query6 329 251 228 228
query7 4722 522 299 299
query8 326 275 250 250
query9 9680 2682 2673 2673
query10 500 341 286 286
query11 15918 15079 14876 14876
query12 174 116 111 111
query13 1672 551 424 424
query14 11216 9215 9122 9122
query15 215 197 203 197
query16 7564 687 516 516
query17 1210 735 629 629
query18 2036 421 329 329
query19 204 216 170 170
query20 131 126 125 125
query21 220 131 115 115
query22 4267 4330 4217 4217
query23 34042 32951 32693 32693
query24 8550 2428 2454 2428
query25 594 527 489 489
query26 1243 280 169 169
query27 2742 509 364 364
query28 4368 2210 2188 2188
query29 833 590 477 477
query30 293 230 202 202
query31 926 818 709 709
query32 78 73 73 73
query33 576 365 325 325
query34 817 912 532 532
query35 799 834 750 750
query36 940 1011 896 896
query37 116 107 82 82
query38 3494 3533 3530 3530
query39 1505 1420 1443 1420
query40 218 129 121 121
query41 64 64 69 64
query42 126 118 115 115
query43 500 492 462 462
query44 1353 838 837 837
query45 188 180 173 173
query46 852 1021 664 664
query47 1785 1822 1734 1734
query48 386 417 325 325
query49 748 505 417 417
query50 649 684 403 403
query51 4063 3924 3897 3897
query52 121 115 104 104
query53 236 279 205 205
query54 593 590 533 533
query55 88 86 87 86
query56 345 328 309 309
query57 1174 1191 1131 1131
query58 274 277 257 257
query59 2561 2596 2570 2570
query60 354 336 376 336
query61 162 160 162 160
query62 811 723 676 676
query63 229 199 199 199
query64 4407 1132 813 813
query65 4037 3950 3955 3950
query66 1107 438 391 391
query67 15522 15277 15353 15277
query68 7850 929 581 581
query69 499 325 276 276
query70 1285 1314 1234 1234
query71 560 344 312 312
query72 5505 4980 4950 4950
query73 703 605 373 373
query74 8841 9120 8639 8639
query75 3990 3332 2847 2847
query76 3642 1206 736 736
query77 814 426 315 315
query78 9472 9616 8943 8943
query79 1560 802 583 583
query80 617 563 496 496
query81 492 261 250 250
query82 237 158 132 132
query83 265 267 258 258
query84 251 111 93 93
query85 954 480 427 427
query86 329 303 291 291
query87 3724 3766 3661 3661
query88 3216 2253 2249 2249
query89 395 338 308 308
query90 2025 223 219 219
query91 163 167 131 131
query92 75 71 65 65
query93 1633 1020 644 644
query94 641 445 338 338
query95 396 317 308 308
query96 498 561 280 280
query97 2887 2978 2868 2868
query98 245 218 214 214
query99 1344 1439 1291 1291
Total cold run time: 275475 ms
Total hot run time: 188762 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #56117:
URL: https://github.com/apache/doris/pull/56117#discussion_r2366626252
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -286,7 +287,26 @@ public List<Split> getSplits(int numBackends) throws UserException {
// And for counting the number of selected partitions for this paimon table.
Map<BinaryRow, Map<String, String>> partitionInfoMaps = new HashMap<>();
// if applyCountPushdown is true, we can't split the DataSplit
- long realFileSplitSize = getRealFileSplitSize(applyCountPushdown ? Long.MAX_VALUE : 0);
+ long realFileSplitSize = applyCountPushdown ? Long.MAX_VALUE : getRealFileSplitSize(0);
+
+ long maxExternalSplitNum = sessionVariable.getMaxExternalSplitNum();
+ Preconditions.checkState(maxExternalSplitNum > 0,
+ "max_external_split_num must be greater than 0, but got: " + maxExternalSplitNum);
+ long totalRawFileSize = dataSplits.stream()
+ .mapToLong(split -> split.rawConvertible()
+ ? split.convertToRawFiles().get().stream().mapToLong(RawFile::fileSize).sum()
+ : 0)
+ .sum();
+ if (totalRawFileSize > 0 && realFileSplitSize > 0) {
+ long estimatedSplitNum = totalRawFileSize / realFileSplitSize;
+ if (estimatedSplitNum > maxExternalSplitNum) {
+ realFileSplitSize = totalRawFileSize / maxExternalSplitNum;
+ LOG.info("The estimated split num is {} which exceeds the limit {}, "
Review Comment:
use debug
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -286,7 +287,26 @@ public List<Split> getSplits(int numBackends) throws UserException {
// And for counting the number of selected partitions for this paimon table.
Map<BinaryRow, Map<String, String>> partitionInfoMaps = new HashMap<>();
// if applyCountPushdown is true, we can't split the DataSplit
- long realFileSplitSize = getRealFileSplitSize(applyCountPushdown ? Long.MAX_VALUE : 0);
+ long realFileSplitSize = applyCountPushdown ? Long.MAX_VALUE : getRealFileSplitSize(0);
+
+ long maxExternalSplitNum = sessionVariable.getMaxExternalSplitNum();
+ Preconditions.checkState(maxExternalSplitNum > 0,
Review Comment:
Move this check to session variable's `checker` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3306393635
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3311959058
# FE Regression Coverage Report
Increment line coverage ` 61.90% (13/21)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56117_534c519a993dfd81f89723eac91b956862134ebf_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56117_534c519a993dfd81f89723eac91b956862134ebf_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3300879145
<details>
<summary>TPC-H: <b>Total hot run time: 34557 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 006aefed780078304a28501c31e8a98605c031b3, data reload: false
------ Round 1 ----------------------------------
q1 17638 5187 5054 5054
q2 2018 321 203 203
q3 10280 1294 713 713
q4 10235 1013 526 526
q5 7533 2416 2364 2364
q6 185 166 136 136
q7 937 775 650 650
q8 9359 1329 1111 1111
q9 6941 5103 5199 5103
q10 6942 2350 1969 1969
q11 500 313 280 280
q12 356 363 228 228
q13 17798 3626 3041 3041
q14 238 240 218 218
q15 567 499 500 499
q16 994 1001 932 932
q17 609 859 371 371
q18 7427 7082 7075 7075
q19 1424 969 562 562
q20 343 356 224 224
q21 3713 2574 2351 2351
q22 1039 1053 947 947
Total cold run time: 107076 ms
Total hot run time: 34557 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5236 5111 5173 5111
q2 265 331 224 224
q3 2201 2675 2262 2262
q4 1342 1791 1346 1346
q5 4257 4503 4538 4503
q6 219 168 133 133
q7 2021 1993 1793 1793
q8 2662 2544 2609 2544
q9 7391 7275 7302 7275
q10 3110 3392 2819 2819
q11 602 542 511 511
q12 701 793 644 644
q13 3722 3842 3364 3364
q14 288 309 278 278
q15 561 478 476 476
q16 1194 1099 1068 1068
q17 1226 1524 1454 1454
q18 7899 7953 7506 7506
q19 827 832 1018 832
q20 2021 2037 1895 1895
q21 4830 4436 4281 4281
q22 1120 1034 994 994
Total cold run time: 53695 ms
Total hot run time: 51313 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3301229820
# FE Regression Coverage Report
Increment line coverage ` 91.67% (11/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56117_006aefed780078304a28501c31e8a98605c031b3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56117_006aefed780078304a28501c31e8a98605c031b3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3307563639
<details>
<summary>TPC-DS: <b>Total hot run time: 2779 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a1337bc0b5c13c6ffc2ec7d93f872c64b0ed399, data reload: false
query1 1093 20 12 12
query2 7125 17 14 14
query3 7644 13 12 12
query4 26275 12 10 10
query5 4439 14 12 12
query6 383 12 12 12
query7 5434 11 11 11
query8 372 19 18 18
query9 9199 12 10 10
query10 728 11 11 11
query11 16182 12 11 11
query12 183 11 10 10
query13 1718 11 9 9
query14 10808 15 14 14
query15 219 12 11 11
query16 7180 11 10 10
query17 1532 12 11 11
query18 2802 13 11 11
query19 213 11 9 9
query20 133 11 11 11
query21 228 11 11 11
query22 4217 10 9 9
query23 33664 17 13 13
query24 10273 14 12 12
query25 775 12 11 11
query26 1651 11 10 10
query27 3391 11 9 9
query28 6359 11 9 9
query29 1747 11 11 11
query30 627 12 11 11
query31 1746 11 10 10
query32 118 12 10 10
query33 1337 11 9 9
query34 1689 863 536 536
query35 909 11 14 11
query36 1004 11 11 11
query37 139 11 10 10
query38 3479 10 9 9
query39 1552 737 762 737
query40 312 11 10 10
query41 84 12 11 11
query42 146 11 10 10
query43 485 10 9 9
query44 1369 10 9 9
query45 198 10 10 10
query46 904 10 10 10
query47 1801 11 9 9
query48 416 11 10 10
query49 1229 12 10 10
query50 778 11 11 11
query51 3953 10 9 9
query52 122 12 10 10
query53 251 11 11 11
query54 830 10 9 9
query55 95 10 9 9
query56 331 10 9 9
query57 1243 10 9 9
query58 305 12 9 9
query59 2611 9 8 8
query60 376 10 8 8
query61 174 10 9 9
query62 835 9 9 9
query63 254 10 9 9
query64 5523 10 9 9
query65 3997 9 9 9
query66 1581 11 87 11
query67 17431 34 8 8
query68 3470 10 9 9
query69 541 11 11 11
query70 1392 10 10 10
query71 447 352 344 344
query72 6974 10 9 9
query73 705 10 9 9
query74 9889 10 9 9
query75 3400 10 9 9
query76 2586 9 11 9
query77 1080 12 10 10
query78 9853 11 10 10
query79 1479 9 10 9
query80 784 12 10 10
query81 1634 10 9 9
query82 232 10 9 9
query83 286 9 9 9
query84 318 10 11 10
query85 1393 9 10 9
query86 765 9 8 8
query87 3878 9 9 9
query88 2811 10 8 8
query89 404 10 9 9
query90 2163 10 9 9
query91 168 9 10 9
query92 91 13 11 11
query93 1511 9 9 9
query94 2743 9 10 9
query95 514 11 9 9
query96 425 10 9 9
query97 3046 10 9 9
query98 240 217 239 217
query99 1470 9 8 8
Total cold run time: 292893 ms
Total hot run time: 2779 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 closed pull request #56117: [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables
URL: https://github.com/apache/doris/pull/56117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3298728163
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 closed pull request #56117: [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables
URL: https://github.com/apache/doris/pull/56117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3298748843
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3307424843
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3307534326
<details>
<summary>TPC-H: <b>Total hot run time: 1477 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a1337bc0b5c13c6ffc2ec7d93f872c64b0ed399, data reload: false
------ Round 1 ----------------------------------
q1 17647 29 19 19
q2 2629 28 29 28
q3 10702 16 15 15
q4 11150 16 15 15
q5 8096 16 14 14
q6 227 14 14 14
q7 933 22 22 22
q8 9449 15 14 14
q9 7282 15 13 13
q10 6960 12 11 11
q11 654 22 20 20
q12 355 12 11 11
q13 18623 12 11 11
q14 256 13 11 11
q15 549 14 11 11
q16 1040 1001 938 938
q17 576 12 11 11
q18 7722 11 11 11
q19 1134 13 12 12
q20 1960 367 244 244
q21 9122 21 21 21
q22 1102 12 11 11
Total cold run time: 118168 ms
Total hot run time: 1477 ms
----- Round 2, with runtime_filter_mode=off -----
q1 11 11 11 11
q2 20 21 21 21
q3 12 12 11 11
q4 13 12 11 11
q5 12 10 13 10
q6 10 11 10 10
q7 19 19 17 17
q8 10 10 11 10
q9 10 10 10 10
q10 9 10 10 10
q11 18 20 20 20
q12 11 10 10 10
q13 10 10 10 10
q14 10 10 10 10
q15 10 10 9 9
q16 1033 1086 1020 1020
q17 14 11 11 11
q18 10 12 10 10
q19 11 11 11 11
q20 1871 1945 1838 1838
q21 20 19 19 19
q22 9 9 9 9
Total cold run time: 3153 ms
Total hot run time: 3098 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](multi-catalog) Avoid over-splitting small files when file count is sufficient for parallelism [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3300824531
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3307584268
<details>
<summary>ClickBench: <b>Total hot run time: 0.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a1337bc0b5c13c6ffc2ec7d93f872c64b0ed399, data reload: false
query1 0.06 0.02 0.01
query2 0.10 0.00 0.01
query3 0.27 0.01 0.00
query4 1.76 0.01 0.01
query5 0.29 0.00 0.01
query6 1.66 0.01 0.01
query7 0.05 0.01 0.00
query8 0.06 0.00 0.00
query9 0.65 0.01 0.00
query10 0.60 0.00 0.01
query11 0.18 0.00 0.00
query12 0.16 0.01 0.01
query13 0.63 0.00 0.00
query14 1.06 0.00 0.00
query15 0.87 0.00 0.01
query16 0.41 0.01 0.00
query17 1.08 0.00 0.00
query18 0.22 0.00 0.00
query19 1.96 0.01 0.01
query20 0.03 0.01 0.00
query21 15.37 0.00 0.00
query22 5.74 0.01 0.00
query23 15.78 0.00 0.01
query24 1.36 0.00 0.00
query25 0.21 0.00 0.00
query26 0.17 0.00 0.00
query27 0.12 0.00 0.00
query28 1.06 0.01 0.00
query29 13.28 0.01 0.01
query30 0.30 0.00 0.01
query31 2.31 0.00 0.00
query32 5.78 0.01 0.00
query33 4.35 0.01 0.00
query34 7.37 0.01 0.00
query35 6.38 0.00 0.00
query36 0.67 0.00 0.01
query37 0.11 0.00 0.00
query38 0.08 0.01 0.00
query39 0.05 0.00 0.00
query40 0.20 0.00 0.00
query41 0.10 0.00 0.00
query42 0.07 0.00 0.00
query43 0.05 0.00 0.00
Total cold run time: 93.01 s
Total hot run time: 0.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](paimon) Intro SessionVariable max_external_split_num to limit the maximum number of file splits when querying external tables [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #56117:
URL: https://github.com/apache/doris/pull/56117#issuecomment-3311445601
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org