You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2024/01/02 06:38:34 UTC
[PR] [improvement](scan) make global runtime filter support in-list filter [doris]
mrhhsg opened a new pull request, #29394:
URL: https://github.com/apache/doris/pull/29394
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874792169
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873679104
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 7defdb6ebd8b61979ac6849fe87458b1b62f6038, data reload: false
------ Round 1 ----------------------------------
q1 18132 5432 5161 5161
q2 2027 156 148 148
q3 10610 1079 1137 1079
q4 10228 744 821 744
q5 7868 2991 2911 2911
q6 215 135 134 134
q7 928 509 539 509
q8 9255 2047 2080 2047
q9 6915 6492 6463 6463
q10 8276 3048 3050 3048
q11 431 208 210 208
q12 394 241 236 236
q13 18001 3654 3615 3615
q14 250 216 212 212
q15 578 550 531 531
q16 459 415 394 394
q17 981 512 505 505
q18 7490 6679 6682 6679
q19 1597 1379 1334 1334
q20 679 348 342 342
q21 2893 2345 2459 2345
q22 372 318 317 317
Total cold run time: 108579 ms
Total hot run time: 38962 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5145 5109 5131 5109
q2 341 240 271 240
q3 3319 3304 3252 3252
q4 2138 2005 1976 1976
q5 5808 5811 5812 5811
q6 218 127 125 125
q7 2315 1927 1891 1891
q8 3408 3471 3541 3471
q9 8850 8764 8749 8749
q10 3829 3831 3869 3831
q11 579 491 505 491
q12 821 664 654 654
q13 6672 3181 3182 3181
q14 302 259 266 259
q15 597 525 532 525
q16 543 518 493 493
q17 1964 1794 1792 1792
q18 8709 8463 8356 8356
q19 1652 1629 1604 1604
q20 2205 1982 1949 1949
q21 5635 5361 5349 5349
q22 536 464 502 464
Total cold run time: 65586 ms
Total hot run time: 59572 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874864497
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 4266756f6053f33ddcf3398a55a1aab358fab1b2, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 922 343 333 333
query2 6450 1897 1954 1897
query3 6643 201 200 200
query4 26912 22537 22447 22447
query5 4377 498 548 498
query6 276 182 177 177
query7 4577 277 264 264
query8 226 195 192 192
query9 8208 2562 2565 2562
query10 383 256 258 256
query11 16222 15677 15634 15634
query12 134 79 71 71
query13 1645 320 324 320
query14 11997 6992 7090 6992
query15 234 189 185 185
query16 6499 269 271 269
query17 1832 516 502 502
query18 1946 286 262 262
query19 216 145 137 137
query20 83 78 76 76
query21 188 97 96 96
query22 5009 4701 4796 4701
query23 31895 31177 31290 31177
query24 11840 2786 2757 2757
query25 581 354 352 352
query26 1690 141 145 141
query27 2865 271 267 267
query28 7179 1949 1952 1949
query29 2054 381 387 381
query30 287 141 143 141
query31 987 764 778 764
query32 89 58 57 57
query33 731 283 264 264
query34 886 437 436 436
query35 955 758 707 707
query36 1338 1198 1284 1198
query37 186 73 72 72
query38 3404 3299 3264 3264
query39 1316 1276 1275 1275
query40 301 89 94 89
query41 37 36 37 36
query42 91 89 89 89
query43 516 494 495 494
query44 1119 714 721 714
query45 198 185 181 181
query46 1074 642 641 641
query47 1658 1515 1559 1515
query48 324 259 262 259
query49 1212 332 323 323
query50 750 323 329 323
query51 5382 5253 5291 5253
query52 88 89 84 84
query53 214 147 143 143
query54 1422 570 567 567
query55 97 88 88 88
query56 210 200 200 200
query57 998 957 951 951
query58 224 202 198 198
query59 2673 2557 2598 2557
query60 255 240 231 231
query61 89 87 87 87
query62 652 434 459 434
query63 162 147 156 147
query64 5899 1742 1668 1668
query65 3328 3264 3239 3239
query66 1384 344 329 329
query67 15481 15327 15293 15293
query68 12467 510 523 510
query69 521 255 253 253
query70 1686 1490 1533 1490
query71 496 220 216 216
query72 5733 3618 3621 3618
query73 2905 314 314 314
query74 7006 6338 6405 6338
query75 5087 2328 2275 2275
query76 6254 1131 1160 1131
query77 653 244 251 244
query78 9109 8879 8529 8529
query79 2360 498 505 498
query80 685 369 356 356
query81 461 201 207 201
query82 212 100 108 100
query83 166 131 134 131
query84 245 53 51 51
query85 1006 297 290 290
query86 402 395 414 395
query87 3601 3443 3374 3374
query88 3481 2279 2294 2279
query89 337 253 254 253
query90 1937 214 213 213
query91 124 94 93 93
query92 59 53 55 53
query93 2106 495 502 495
query94 799 184 192 184
query95 476 427 405 405
query96 633 317 324 317
query97 4270 4163 4151 4151
query98 208 189 192 189
query99 1139 840 828 828
Total cold run time: 295724 ms
Total hot run time: 179180 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874867653
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 4266756f6053f33ddcf3398a55a1aab358fab1b2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5593 5188 5210 5188
q2 396 182 152 152
q3 1452 1192 1236 1192
q4 1090 864 760 760
q5 3177 3172 3100 3100
q6 241 136 134 134
q7 993 569 513 513
q8 2153 2265 2247 2247
q9 6714 6709 6704 6704
q10 3163 3132 3185 3132
q11 340 215 232 215
q12 387 242 241 241
q13 4475 3716 3673 3673
q14 258 214 213 213
q15 607 548 537 537
q16 464 402 403 402
q17 1046 493 482 482
q18 7172 6799 6902 6799
q19 1652 1562 1458 1458
q20 567 339 355 339
q21 2897 2521 2535 2521
q22 399 319 326 319
Total cold run time: 45236 ms
Total hot run time: 40321 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5187 5117 5145 5117
q2 342 243 256 243
q3 3392 3331 3331 3331
q4 2157 2057 2044 2044
q5 5986 5972 6024 5972
q6 231 127 128 127
q7 2419 1954 1970 1954
q8 3581 3693 3695 3693
q9 9119 9009 9080 9009
q10 3939 3953 3953 3953
q11 594 494 490 490
q12 800 652 672 652
q13 3923 3225 3198 3198
q14 312 288 268 268
q15 617 560 552 552
q16 564 502 495 495
q17 2072 1813 1836 1813
q18 8820 8395 8378 8378
q19 1740 1699 1707 1699
q20 2286 2024 2018 2018
q21 5740 5379 5388 5379
q22 581 539 490 490
Total cold run time: 64402 ms
Total hot run time: 60875 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875223096
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874843057
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875280759
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 5ea45c5dbaa530cfef408482160980ae4aa10a21, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 938 346 341 341
query2 6423 1933 1967 1933
query3 6665 202 204 202
query4 28341 22489 22377 22377
query5 4079 541 518 518
query6 258 172 172 172
query7 4580 260 265 260
query8 224 209 198 198
query9 8261 2567 2737 2567
query10 435 247 239 239
query11 16333 15548 15566 15548
query12 141 82 72 72
query13 1637 322 320 320
query14 12113 7048 7275 7048
query15 232 189 192 189
query16 6506 267 254 254
query17 1873 499 489 489
query18 1959 270 258 258
query19 273 137 137 137
query20 81 79 76 76
query21 184 103 93 93
query22 5040 4818 4797 4797
query23 32121 30965 31156 30965
query24 11967 2838 2777 2777
query25 579 344 332 332
query26 1686 147 144 144
query27 2855 272 280 272
query28 6926 1974 1970 1970
query29 2035 379 376 376
query30 292 146 142 142
query31 983 789 772 772
query32 89 61 58 58
query33 741 278 250 250
query34 930 447 445 445
query35 880 704 739 704
query36 1350 1241 1263 1241
query37 193 74 78 74
query38 3375 3272 3235 3235
query39 1348 1287 1284 1284
query40 301 89 93 89
query41 38 35 34 34
query42 98 92 92 92
query43 534 476 483 476
query44 1070 710 723 710
query45 195 188 185 185
query46 1086 658 631 631
query47 1752 1657 1456 1456
query48 340 256 263 256
query49 1225 326 334 326
query50 750 346 362 346
query51 5335 5298 5334 5298
query52 89 88 86 86
query53 203 151 147 147
query54 1365 584 550 550
query55 93 90 86 86
query56 219 202 197 197
query57 1056 898 909 898
query58 229 199 198 198
query59 2829 2613 2502 2502
query60 255 243 228 228
query61 81 80 82 80
query62 649 473 452 452
query63 171 146 147 146
query64 5855 1713 1696 1696
query65 3356 3266 3289 3266
query66 1387 340 348 340
query67 15773 15588 15107 15107
query68 12613 533 545 533
query69 542 248 251 248
query70 1674 1407 1543 1407
query71 501 218 223 218
query72 5552 3532 3464 3464
query73 3024 321 304 304
query74 7024 6418 6410 6410
query75 5284 2279 2260 2260
query76 6348 1072 1130 1072
query77 662 257 264 257
query78 9215 8620 8774 8620
query79 1535 509 506 506
query80 554 359 355 355
query81 461 208 210 208
query82 205 97 104 97
query83 194 135 132 132
query84 242 52 53 52
query85 960 275 271 271
query86 389 421 382 382
query87 3602 3349 3399 3349
query88 3301 2276 2268 2268
query89 347 264 276 264
query90 1909 210 210 210
query91 130 88 89 88
query92 63 55 54 54
query93 1804 445 440 440
query94 894 183 186 183
query95 482 403 414 403
query96 637 321 319 319
query97 4304 4152 4144 4144
query98 218 202 190 190
query99 1135 805 789 789
Total cold run time: 297065 ms
Total hot run time: 178504 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874792154
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873663329
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29394:
URL: https://github.com/apache/doris/pull/29394#discussion_r1439186877
##########
be/src/exprs/runtime_filter.cpp:
##########
@@ -453,9 +453,11 @@ class RuntimePredicateWrapper {
std::vector<vectorized::VExprSPtr>& push_exprs, const TExpr& probe_expr);
Status merge(const RuntimePredicateWrapper* wrapper) {
Review Comment:
warning: function 'merge' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status merge(const RuntimePredicateWrapper* wrapper) {
^
```
<details>
<summary>Additional context</summary>
**be/src/exprs/runtime_filter.cpp:456:** +1
```cpp
_filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER &&
^
```
**be/src/exprs/runtime_filter.cpp:461:** +1
```cpp
bool can_not_merge_other = _filter_type != RuntimeFilterType::IN_OR_BLOOM_FILTER &&
^
```
**be/src/exprs/runtime_filter.cpp:464:** +1
```cpp
CHECK(!can_not_merge_in_or_bloom && !can_not_merge_other)
^
```
**be/src/exprs/runtime_filter.cpp:469:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
switch (_filter_type) {
^
```
**be/src/exprs/runtime_filter.cpp:471:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_is_ignored_in_filter) {
^
```
**be/src/exprs/runtime_filter.cpp:473:** +1, nesting level increased to 2
```cpp
} else if (wrapper->_is_ignored_in_filter) {
^
```
**be/src/exprs/runtime_filter.cpp:485:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (_max_in_num >= 0 && _context.hybrid_set->size() >= _max_in_num) {
^
```
**be/src/exprs/runtime_filter.cpp:505:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/exprs/runtime_filter.cpp:505:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/exprs/runtime_filter.cpp:510:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/exprs/runtime_filter.cpp:510:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/exprs/runtime_filter.cpp:515:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
auto real_filter_type = _is_bloomfilter ? RuntimeFilterType::BLOOM_FILTER
^
```
**be/src/exprs/runtime_filter.cpp:519:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (other_filter_type == RuntimeFilterType::IN_OR_BLOOM_FILTER) {
^
```
**be/src/exprs/runtime_filter.cpp:520:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
other_filter_type = wrapper->_is_bloomfilter ? RuntimeFilterType::BLOOM_FILTER
^
```
**be/src/exprs/runtime_filter.cpp:524:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (real_filter_type == RuntimeFilterType::IN_FILTER) {
^
```
**be/src/exprs/runtime_filter.cpp:525:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (other_filter_type == RuntimeFilterType::IN_FILTER) { // in merge in
^
```
**be/src/exprs/runtime_filter.cpp:531:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (_max_in_num >= 0 && _context.hybrid_set->size() >= _max_in_num) {
^
```
**be/src/exprs/runtime_filter.cpp:537:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/exprs/runtime_filter.cpp:541:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_context.bloom_filter_func->merge(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/exprs/runtime_filter.cpp:541:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_context.bloom_filter_func->merge(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/exprs/runtime_filter.cpp:544:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/exprs/runtime_filter.cpp:545:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (other_filter_type == RuntimeFilterType::IN_FILTER) { // bloom filter merge in
^
```
**be/src/exprs/runtime_filter.cpp:552:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/exprs/runtime_filter.cpp:553:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_context.bloom_filter_func->merge(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/exprs/runtime_filter.cpp:553:** +5, including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(_context.bloom_filter_func->merge(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874871295
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.25 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17184109387 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875283829
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17183726812 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875523422
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875267577
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 5ea45c5dbaa530cfef408482160980ae4aa10a21, data reload: false
------ Round 1 ----------------------------------
q1 17980 5152 5162 5152
q2 2045 160 144 144
q3 10617 1112 1182 1112
q4 10232 775 806 775
q5 7815 2975 2904 2904
q6 217 135 133 133
q7 908 513 529 513
q8 9316 2058 2031 2031
q9 6857 6424 6388 6388
q10 8245 3072 3056 3056
q11 452 228 210 210
q12 392 237 235 235
q13 18001 3637 3625 3625
q14 238 221 216 216
q15 580 522 547 522
q16 449 398 394 394
q17 984 551 501 501
q18 7601 6794 6764 6764
q19 1616 1319 1368 1319
q20 698 340 332 332
q21 2828 2366 2479 2366
q22 391 326 344 326
Total cold run time: 108462 ms
Total hot run time: 39018 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5247 5140 5152 5140
q2 341 231 248 231
q3 3315 3294 3270 3270
q4 2117 2003 2012 2003
q5 5815 5771 5812 5771
q6 218 124 122 122
q7 2306 1879 1892 1879
q8 3381 3471 3505 3471
q9 8839 8735 8779 8735
q10 3796 3844 3850 3844
q11 598 487 511 487
q12 804 650 638 638
q13 7239 3238 3180 3180
q14 305 266 264 264
q15 600 525 538 525
q16 569 525 513 513
q17 1939 1766 1761 1761
q18 8753 8418 8344 8344
q19 1629 1614 1598 1598
q20 2244 1987 1944 1944
q21 5530 5271 5294 5271
q22 520 480 489 480
Total cold run time: 66105 ms
Total hot run time: 59471 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874856553
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 4266756f6053f33ddcf3398a55a1aab358fab1b2, data reload: false
------ Round 1 ----------------------------------
q1 17661 5165 5095 5095
q2 2018 153 141 141
q3 10542 1141 1186 1141
q4 10190 774 792 774
q5 7799 3026 2935 2935
q6 209 134 131 131
q7 905 553 501 501
q8 9290 2009 2011 2009
q9 6838 6382 6343 6343
q10 8255 3021 2990 2990
q11 440 230 210 210
q12 385 227 230 227
q13 17997 3662 3620 3620
q14 241 223 211 211
q15 593 537 534 534
q16 450 426 413 413
q17 956 459 535 459
q18 7329 6708 6625 6625
q19 1591 1287 1457 1287
q20 727 328 355 328
q21 2788 2350 2410 2350
q22 385 330 330 330
Total cold run time: 107589 ms
Total hot run time: 38654 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5176 5027 5078 5027
q2 337 254 248 248
q3 3288 3241 3207 3207
q4 2078 1964 1942 1942
q5 5760 5771 5746 5746
q6 211 126 123 123
q7 2299 1891 1893 1891
q8 3368 3422 3464 3422
q9 8795 8685 8710 8685
q10 3752 3832 3841 3832
q11 562 482 481 481
q12 780 646 640 640
q13 8492 3215 3187 3187
q14 287 281 282 281
q15 590 538 538 538
q16 540 516 484 484
q17 1922 1745 1717 1717
q18 8602 8282 8248 8248
q19 1614 1561 1567 1561
q20 2186 1957 1950 1950
q21 5588 5257 5199 5199
q22 542 469 469 469
Total cold run time: 66769 ms
Total hot run time: 58878 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873696182
TeamCity be ut coverage result:
Function Coverage: 36.63% (8613/23515)
Line Coverage: 28.68% (69989/244065)
Region Coverage: 27.67% (36231/130957)
Branch Coverage: 24.36% (18511/75980)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7defdb6ebd8b61979ac6849fe87458b1b62f6038_7defdb6ebd8b61979ac6849fe87458b1b62f6038/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873687357
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.02 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.6 seconds inserted 10000000 Rows, about 362K ops/s
storage size: 17183912520 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #29394:
URL: https://github.com/apache/doris/pull/29394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875272818
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 5ea45c5dbaa530cfef408482160980ae4aa10a21, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5509 5164 5154 5154
q2 401 172 148 148
q3 1469 1173 1119 1119
q4 1108 857 831 831
q5 3135 3143 3051 3051
q6 225 148 130 130
q7 989 583 529 529
q8 2159 2229 2286 2229
q9 6707 6666 6680 6666
q10 3214 3101 3103 3101
q11 355 212 210 210
q12 379 229 239 229
q13 4378 3674 3684 3674
q14 252 228 220 220
q15 600 551 549 549
q16 455 425 400 400
q17 1041 513 567 513
q18 7078 6772 7097 6772
q19 1641 1518 1473 1473
q20 587 334 353 334
q21 2887 2559 2519 2519
q22 397 338 329 329
Total cold run time: 44966 ms
Total hot run time: 40180 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5160 5091 5151 5091
q2 344 229 247 229
q3 3358 3320 3317 3317
q4 2173 1976 1999 1976
q5 5934 5911 5924 5911
q6 229 124 124 124
q7 2388 1932 1985 1932
q8 3574 3668 3700 3668
q9 9041 9018 8971 8971
q10 3872 3900 3895 3895
q11 583 531 495 495
q12 820 620 623 620
q13 3889 3211 3193 3193
q14 309 269 272 269
q15 606 542 545 542
q16 561 520 510 510
q17 2034 1830 1819 1819
q18 8705 8347 8402 8347
q19 1745 1663 1673 1663
q20 2288 1994 1998 1994
q21 5725 5436 5308 5308
q22 575 536 507 507
Total cold run time: 63913 ms
Total hot run time: 60381 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873686100
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 7defdb6ebd8b61979ac6849fe87458b1b62f6038, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 943 362 339 339
query2 6418 1916 1877 1877
query3 6645 202 203 202
query4 26898 22572 22427 22427
query5 4351 505 528 505
query6 270 184 178 178
query7 4577 272 265 265
query8 230 213 210 210
query9 8277 2531 2743 2531
query10 433 251 261 251
query11 16211 15495 15562 15495
query12 132 81 73 73
query13 1632 324 329 324
query14 11634 7058 7094 7058
query15 242 182 189 182
query16 6502 277 271 271
query17 1847 504 491 491
query18 1945 266 260 260
query19 278 141 137 137
query20 81 80 77 77
query21 183 98 93 93
query22 5025 4767 4756 4756
query23 32095 31160 31082 31082
query24 12151 2815 2810 2810
query25 589 355 352 352
query26 1679 142 142 142
query27 2904 279 282 279
query28 7142 1962 1959 1959
query29 2054 391 384 384
query30 286 143 149 143
query31 984 787 787 787
query32 87 61 56 56
query33 735 276 273 273
query34 875 441 433 433
query35 877 772 777 772
query36 1285 1159 1276 1159
query37 181 74 74 74
query38 3356 3276 3305 3276
query39 1321 1287 1282 1282
query40 300 88 92 88
query41 36 35 36 35
query42 93 92 92 92
query43 527 468 502 468
query44 1049 722 731 722
query45 195 184 184 184
query46 1068 636 642 636
query47 1740 1518 1601 1518
query48 338 261 266 261
query49 1190 334 325 325
query50 742 322 318 318
query51 5448 5262 5342 5262
query52 93 95 88 88
query53 210 150 149 149
query54 1402 567 573 567
query55 96 90 95 90
query56 205 197 208 197
query57 1064 962 966 962
query58 220 202 203 202
query59 2781 2492 2565 2492
query60 273 231 255 231
query61 89 84 85 84
query62 639 455 471 455
query63 168 149 154 149
query64 5911 1760 1715 1715
query65 3361 3269 3257 3257
query66 1381 332 333 332
query67 15693 15000 15295 15000
query68 12530 545 575 545
query69 521 250 247 247
query70 1645 1474 1482 1474
query71 493 230 221 221
query72 5740 3594 3564 3564
query73 2876 316 310 310
query74 6908 6354 6469 6354
query75 5159 2247 2270 2247
query76 6274 1098 1151 1098
query77 658 277 280 277
query78 9107 8704 8717 8704
query79 1038 507 540 507
query80 544 365 365 365
query81 450 210 210 210
query82 210 98 105 98
query83 178 132 138 132
query84 246 53 53 53
query85 978 285 276 276
query86 385 381 407 381
query87 3533 3356 3323 3323
query88 3055 2273 2270 2270
query89 339 257 261 257
query90 1930 204 199 199
query91 120 90 92 90
query92 60 54 54 54
query93 1424 513 494 494
query94 827 185 178 178
query95 473 428 404 404
query96 635 315 314 314
query97 4289 4188 4208 4188
query98 200 202 192 192
query99 1135 849 798 798
Total cold run time: 293627 ms
Total hot run time: 178968 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1874860378
TeamCity be ut coverage result:
Function Coverage: 36.62% (8615/23524)
Line Coverage: 28.67% (70006/244175)
Region Coverage: 27.65% (36241/131047)
Branch Coverage: 24.35% (18509/76026)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4266756f6053f33ddcf3398a55a1aab358fab1b2_4266756f6053f33ddcf3398a55a1aab358fab1b2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1875278887
TeamCity be ut coverage result:
Function Coverage: 36.62% (8613/23523)
Line Coverage: 28.66% (69977/244168)
Region Coverage: 27.64% (36221/131050)
Branch Coverage: 24.33% (18501/76028)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5ea45c5dbaa530cfef408482160980ae4aa10a21_5ea45c5dbaa530cfef408482160980ae4aa10a21/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](scan) make global runtime filter support in-list filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29394:
URL: https://github.com/apache/doris/pull/29394#issuecomment-1873683077
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 7defdb6ebd8b61979ac6849fe87458b1b62f6038, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5498 5155 5133 5133
q2 398 173 162 162
q3 1465 1167 1161 1161
q4 1089 827 779 779
q5 3107 3131 3078 3078
q6 227 143 132 132
q7 983 525 544 525
q8 2154 2227 2237 2227
q9 6693 6615 6612 6612
q10 3194 3099 3207 3099
q11 360 223 224 223
q12 390 240 244 240
q13 4406 3685 3676 3676
q14 258 225 210 210
q15 610 568 548 548
q16 448 395 396 395
q17 1034 519 480 480
q18 7162 6758 6786 6758
q19 1644 1568 1421 1421
q20 547 359 353 353
q21 2899 2464 2426 2426
q22 387 312 325 312
Total cold run time: 44953 ms
Total hot run time: 39950 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5078 5068 5109 5068
q2 352 246 240 240
q3 3390 3343 3302 3302
q4 2147 2045 2020 2020
q5 5953 5925 5951 5925
q6 227 129 130 129
q7 2381 1952 1909 1909
q8 3560 3662 3668 3662
q9 9050 9023 9000 9000
q10 3863 3946 3968 3946
q11 589 489 487 487
q12 814 654 633 633
q13 3925 3236 3185 3185
q14 307 258 294 258
q15 623 556 547 547
q16 578 516 497 497
q17 2012 1849 1777 1777
q18 8886 8349 8439 8349
q19 1732 1664 1668 1664
q20 2278 1998 1982 1982
q21 5716 5382 5362 5362
q22 553 539 510 510
Total cold run time: 64014 ms
Total hot run time: 60452 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org