You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2024/03/29 02:58:44 UTC
[PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
englefly opened a new pull request, #33010:
URL: https://github.com/apache/doris/pull/33010
## Proposed changes
`select count() from ((select l_linenumber from lineitem) except (select o_orderkey from orders)) T join region on abs(T.l_linenumber) = r_regionkey`
because of ABS and set operator, the runtime filter is not generated.
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #33010:
URL: https://github.com/apache/doris/pull/33010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2031015929
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.2 seconds inserted 10000000 Rows, about 617K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2033378555
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2031013861
<details>
<summary>ClickBench: <b>Total hot run time: 30.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false
query1 0.04 0.04 0.02
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.07 0.07
query5 0.49 0.48 0.48
query6 1.14 0.66 0.65
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.51 0.50
query10 0.58 0.57 0.55
query11 0.15 0.11 0.12
query12 0.14 0.11 0.11
query13 0.60 0.59 0.60
query14 0.76 0.78 0.79
query15 0.85 0.84 0.84
query16 0.36 0.36 0.35
query17 0.96 0.96 0.97
query18 0.25 0.25 0.25
query19 1.83 1.68 1.71
query20 0.02 0.01 0.01
query21 15.40 0.78 0.71
query22 2.88 6.35 2.34
query23 17.89 1.24 1.07
query24 1.82 0.23 0.23
query25 0.14 0.09 0.09
query26 0.31 0.18 0.19
query27 0.09 0.09 0.09
query28 13.44 0.93 0.94
query29 12.52 3.43 3.29
query30 0.27 0.06 0.06
query31 2.86 0.38 0.38
query32 3.27 0.48 0.47
query33 2.79 2.88 2.94
query34 15.50 4.34 4.32
query35 4.37 4.35 4.37
query36 0.68 0.47 0.48
query37 0.22 0.16 0.18
query38 0.16 0.16 0.15
query39 0.05 0.04 0.04
query40 0.18 0.14 0.14
query41 0.10 0.06 0.05
query42 0.06 0.06 0.05
query43 0.04 0.03 0.04
Total cold run time: 105.82 s
Total hot run time: 30.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026529843
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #33010:
URL: https://github.com/apache/doris/pull/33010#discussion_r1548929017
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java:
##########
@@ -497,6 +498,23 @@ public PhysicalRelation visitPhysicalRelation(PhysicalRelation relation, Cascade
return relation;
}
+ @Override
+ public PhysicalSetOperation visitPhysicalSetOperation(PhysicalSetOperation setOperation, CascadesContext context) {
Review Comment:
this is not push down, this is to collect alias map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2033378572
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2031010133
<details>
<summary>TPC-DS: <b>Total hot run time: 183038 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false
query1 880 1122 1124 1122
query2 6273 1899 1878 1878
query3 6665 213 212 212
query4 24911 21445 21531 21445
query5 4205 411 404 404
query6 284 199 194 194
query7 4609 310 293 293
query8 230 190 184 184
query9 8452 2261 2285 2261
query10 451 254 273 254
query11 14875 14500 14596 14500
query12 145 98 96 96
query13 1636 377 387 377
query14 8554 6924 7119 6924
query15 208 178 193 178
query16 6823 286 275 275
query17 968 602 586 586
query18 1836 290 291 290
query19 202 168 163 163
query20 97 95 99 95
query21 194 130 130 130
query22 5045 4886 4872 4872
query23 33386 32966 32640 32640
query24 11315 3254 3205 3205
query25 696 457 438 438
query26 1640 183 166 166
query27 3175 385 423 385
query28 7028 1898 1891 1891
query29 1063 592 617 592
query30 301 162 156 156
query31 1078 746 730 730
query32 106 58 62 58
query33 689 247 244 244
query34 1313 524 518 518
query35 846 755 733 733
query36 1027 855 861 855
query37 154 80 77 77
query38 3683 3577 3665 3577
query39 1651 1614 1594 1594
query40 224 141 149 141
query41 46 48 46 46
query42 116 117 113 113
query43 461 414 419 414
query44 1132 715 714 714
query45 287 264 268 264
query46 1142 858 796 796
query47 2012 1913 1953 1913
query48 408 323 309 309
query49 911 369 365 365
query50 836 409 417 409
query51 6987 6877 6913 6877
query52 118 106 121 106
query53 378 303 304 303
query54 320 256 268 256
query55 89 80 79 79
query56 250 263 235 235
query57 1267 1241 1188 1188
query58 251 233 238 233
query59 2727 2495 2321 2321
query60 266 245 252 245
query61 111 106 106 106
query62 663 463 457 457
query63 310 282 284 282
query64 5067 3474 3482 3474
query65 3037 3032 3036 3032
query66 960 338 322 322
query67 15383 14878 14845 14845
query68 9197 584 584 584
query69 573 332 332 332
query70 1330 1100 1046 1046
query71 525 267 269 267
query72 6380 2581 2402 2402
query73 1407 335 333 333
query74 6696 6393 6445 6393
query75 3739 2326 2327 2326
query76 5588 1130 1194 1130
query77 588 244 244 244
query78 10908 10065 10135 10065
query79 9907 540 532 532
query80 1879 428 413 413
query81 490 227 1067 227
query82 730 104 102 102
query83 199 159 160 159
query84 272 90 92 90
query85 1332 309 281 281
query86 399 276 290 276
query87 3678 3489 3514 3489
query88 3641 2373 2376 2373
query89 546 361 364 361
query90 1922 181 180 180
query91 133 105 104 104
query92 59 49 49 49
query93 6819 524 529 524
query94 1056 194 191 191
query95 438 334 327 327
query96 613 279 270 270
query97 2665 2480 2509 2480
query98 240 218 220 218
query99 1302 833 839 833
Total cold run time: 297704 ms
Total hot run time: 183038 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2030970935
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2031002754
<details>
<summary>TPC-H: <b>Total hot run time: 38491 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false
------ Round 1 ----------------------------------
q1 17629 4054 4053 4053
q2 2007 194 181 181
q3 10471 1219 1388 1219
q4 10210 826 990 826
q5 7454 2959 2912 2912
q6 218 135 131 131
q7 1120 627 605 605
q8 9412 2079 2026 2026
q9 6646 6194 6138 6138
q10 8443 3512 3485 3485
q11 432 244 246 244
q12 380 223 214 214
q13 17781 2865 2924 2865
q14 271 244 244 244
q15 534 491 486 486
q16 525 399 384 384
q17 944 912 875 875
q18 7105 6460 6367 6367
q19 1587 1525 1534 1525
q20 591 323 317 317
q21 3605 3098 3133 3098
q22 365 298 296 296
Total cold run time: 107730 ms
Total hot run time: 38491 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4043 4066 4047 4047
q2 327 221 220 220
q3 2948 2978 2929 2929
q4 1826 1898 1831 1831
q5 5183 5185 5206 5185
q6 207 124 120 120
q7 2211 1796 1839 1796
q8 3204 3251 3255 3251
q9 8471 8406 8456 8406
q10 3751 3901 3987 3901
q11 558 459 463 459
q12 759 555 599 555
q13 16831 3073 3090 3073
q14 305 263 281 263
q15 541 524 503 503
q16 496 434 432 432
q17 1762 1719 1680 1680
q18 8356 7951 7611 7611
q19 1704 1673 1667 1667
q20 2020 1886 1768 1768
q21 5176 4955 4977 4955
q22 516 432 423 423
Total cold run time: 71195 ms
Total hot run time: 55075 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026590656
<details>
<summary>ClickBench: <b>Total hot run time: 31.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false
query1 0.05 0.04 0.03
query2 0.06 0.03 0.03
query3 0.22 0.06 0.06
query4 1.65 0.10 0.10
query5 0.50 0.50 0.52
query6 1.14 0.65 0.65
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.57 0.50 0.49
query10 0.54 0.55 0.55
query11 0.13 0.09 0.10
query12 0.12 0.09 0.09
query13 0.59 0.58 0.57
query14 0.76 0.77 0.80
query15 0.82 0.81 0.82
query16 0.39 0.37 0.37
query17 0.99 0.97 0.97
query18 0.23 0.22 0.22
query19 1.78 1.73 1.70
query20 0.01 0.01 0.01
query21 15.52 0.58 0.57
query22 2.56 2.19 2.15
query23 17.13 0.80 0.73
query24 3.30 2.00 2.48
query25 0.18 0.22 0.15
query26 0.70 0.12 0.11
query27 0.03 0.03 0.06
query28 8.44 0.89 0.86
query29 12.57 3.17 3.22
query30 0.29 0.09 0.08
query31 2.82 0.36 0.36
query32 3.30 0.46 0.45
query33 2.88 2.85 2.85
query34 16.14 4.45 4.40
query35 4.48 4.50 4.45
query36 0.67 0.48 0.49
query37 0.08 0.05 0.06
query38 0.05 0.03 0.03
query39 0.04 0.02 0.02
query40 0.18 0.14 0.14
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 102.11 s
Total hot run time: 31.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026592862
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.2 seconds inserted 10000000 Rows, about 704K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026851232
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026529969
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026889823
<details>
<summary>TPC-DS: <b>Total hot run time: 182039 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false
query1 907 1137 1145 1137
query2 7048 2031 1845 1845
query3 6661 216 211 211
query4 23301 21492 21332 21332
query5 4263 404 385 385
query6 274 180 169 169
query7 4600 304 290 290
query8 231 175 183 175
query9 8499 2283 2259 2259
query10 535 245 255 245
query11 14798 14173 14328 14173
query12 147 93 90 90
query13 1639 372 365 365
query14 9902 6798 6785 6785
query15 250 180 176 176
query16 6997 272 270 270
query17 1458 581 564 564
query18 1807 294 284 284
query19 207 161 155 155
query20 93 93 89 89
query21 202 132 128 128
query22 4979 4821 4842 4821
query23 33377 32890 32991 32890
query24 12411 2818 2801 2801
query25 702 391 390 390
query26 1713 153 157 153
query27 3374 321 321 321
query28 7131 1834 1830 1830
query29 1018 622 605 605
query30 295 151 151 151
query31 994 714 729 714
query32 97 59 55 55
query33 727 261 246 246
query34 1191 479 478 478
query35 814 694 686 686
query36 1054 904 915 904
query37 215 73 68 68
query38 3610 3511 3449 3449
query39 1590 1576 1544 1544
query40 292 140 130 130
query41 50 47 47 47
query42 106 103 97 97
query43 505 460 459 459
query44 1223 734 709 709
query45 271 265 264 264
query46 1086 693 715 693
query47 1936 1832 1832 1832
query48 390 315 298 298
query49 1164 370 372 370
query50 776 385 384 384
query51 6775 6604 6677 6604
query52 106 88 99 88
query53 347 277 278 277
query54 324 233 239 233
query55 88 79 82 79
query56 265 228 226 226
query57 1233 1118 1130 1118
query58 263 211 216 211
query59 2904 2744 2574 2574
query60 270 243 265 243
query61 116 116 117 116
query62 641 446 453 446
query63 309 278 288 278
query64 6581 4144 4117 4117
query65 3133 3050 3020 3020
query66 1495 351 333 333
query67 15397 14994 15035 14994
query68 9412 538 538 538
query69 612 301 313 301
query70 1439 1090 1159 1090
query71 522 279 268 268
query72 7268 2657 2457 2457
query73 1636 324 331 324
query74 6968 6396 6461 6396
query75 4017 2312 2299 2299
query76 5592 923 1046 923
query77 601 262 251 251
query78 10947 10255 10262 10255
query79 9492 546 528 528
query80 1303 430 418 418
query81 502 226 222 222
query82 228 89 94 89
query83 220 172 170 170
query84 265 84 81 81
query85 971 276 266 266
query86 365 299 297 297
query87 3729 3501 3502 3501
query88 5577 2401 2394 2394
query89 529 390 386 386
query90 2047 178 183 178
query91 130 99 98 98
query92 58 50 47 47
query93 6442 534 504 504
query94 1259 183 186 183
query95 403 313 315 313
query96 619 278 265 265
query97 2675 2499 2489 2489
query98 237 225 212 212
query99 1098 847 823 823
Total cold run time: 304320 ms
Total hot run time: 182039 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026897144
<details>
<summary>ClickBench: <b>Total hot run time: 29.4 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.03 0.04
query3 0.23 0.05 0.06
query4 1.65 0.08 0.10
query5 0.50 0.47 0.50
query6 1.15 0.66 0.65
query7 0.02 0.02 0.01
query8 0.06 0.04 0.04
query9 0.55 0.49 0.50
query10 0.55 0.54 0.56
query11 0.16 0.10 0.12
query12 0.14 0.12 0.11
query13 0.58 0.58 0.59
query14 0.76 0.77 0.78
query15 0.83 0.81 0.81
query16 0.36 0.35 0.36
query17 1.01 0.97 1.01
query18 0.23 0.23 0.25
query19 1.92 1.68 1.72
query20 0.02 0.01 0.01
query21 15.55 0.64 0.64
query22 2.46 2.98 1.52
query23 17.06 1.01 0.84
query24 1.17 0.25 0.27
query25 0.10 0.08 0.08
query26 0.22 0.17 0.17
query27 0.08 0.08 0.07
query28 13.94 0.93 0.93
query29 12.52 3.23 3.27
query30 0.28 0.09 0.07
query31 2.82 0.40 0.38
query32 3.25 0.46 0.47
query33 2.84 2.86 2.85
query34 16.80 4.40 4.45
query35 4.48 4.47 4.47
query36 0.61 0.46 0.46
query37 0.18 0.16 0.15
query38 0.15 0.14 0.14
query39 0.05 0.03 0.04
query40 0.20 0.14 0.15
query41 0.09 0.05 0.05
query42 0.05 0.04 0.04
query43 0.04 0.03 0.04
Total cold run time: 105.78 s
Total hot run time: 29.4 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #33010:
URL: https://github.com/apache/doris/pull/33010#discussion_r1548830346
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java:
##########
@@ -497,6 +498,23 @@ public PhysicalRelation visitPhysicalRelation(PhysicalRelation relation, Cascade
return relation;
}
+ @Override
+ public PhysicalSetOperation visitPhysicalSetOperation(PhysicalSetOperation setOperation, CascadesContext context) {
Review Comment:
why not process it in pushDownRuntimeFilter impl of PhysicalSetOperation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026572677
<details>
<summary>TPC-H: <b>Total hot run time: 37306 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false
------ Round 1 ----------------------------------
q1 17595 4700 4098 4098
q2 2104 154 149 149
q3 10585 1117 1187 1117
q4 10228 816 826 816
q5 7469 2928 2891 2891
q6 200 125 125 125
q7 1044 584 567 567
q8 9336 1988 1986 1986
q9 7063 6287 6300 6287
q10 8420 3418 3590 3418
q11 437 239 226 226
q12 405 202 197 197
q13 17793 2873 2821 2821
q14 248 200 197 197
q15 508 463 459 459
q16 479 368 366 366
q17 952 523 592 523
q18 7100 6393 6293 6293
q19 1928 1459 1359 1359
q20 555 247 241 241
q21 3573 2885 2961 2885
q22 331 285 295 285
Total cold run time: 108353 ms
Total hot run time: 37306 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4100 4073 4077 4073
q2 331 232 231 231
q3 2953 2844 2838 2838
q4 1829 1554 1570 1554
q5 5232 5389 5217 5217
q6 197 121 121 121
q7 2183 1795 1802 1795
q8 3152 3274 3283 3274
q9 8403 8392 8408 8392
q10 3755 3736 4658 3736
q11 571 470 459 459
q12 750 582 555 555
q13 17044 3018 3068 3018
q14 294 261 285 261
q15 536 492 483 483
q16 494 498 476 476
q17 1807 1573 1542 1542
q18 7901 7690 7640 7640
q19 1665 1594 1570 1570
q20 1997 1827 1831 1827
q21 4806 4870 4898 4870
q22 543 442 437 437
Total cold run time: 70543 ms
Total hot run time: 54369 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026585413
<details>
<summary>TPC-DS: <b>Total hot run time: 182194 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false
query1 931 351 339 339
query2 6335 2138 1888 1888
query3 6697 217 213 213
query4 31520 21321 21292 21292
query5 4290 408 400 400
query6 288 197 177 177
query7 4631 294 292 292
query8 229 168 171 168
query9 9466 2297 2271 2271
query10 445 257 260 257
query11 14783 14227 14245 14227
query12 134 90 90 90
query13 1612 419 417 417
query14 8917 8036 7936 7936
query15 253 181 177 177
query16 8251 264 265 264
query17 1966 574 558 558
query18 2110 290 300 290
query19 347 155 164 155
query20 95 92 85 85
query21 197 127 126 126
query22 4992 4866 4777 4777
query23 33470 32822 33137 32822
query24 10897 2978 3015 2978
query25 628 418 414 414
query26 1177 168 173 168
query27 2796 372 388 372
query28 7325 1958 1970 1958
query29 923 648 626 626
query30 317 157 160 157
query31 1004 761 767 761
query32 95 58 59 58
query33 790 268 265 265
query34 1232 496 497 496
query35 912 708 725 708
query36 1056 912 924 912
query37 123 69 66 66
query38 3782 3680 3643 3643
query39 1084 1041 1061 1041
query40 183 104 109 104
query41 49 52 48 48
query42 108 96 104 96
query43 505 471 450 450
query44 1173 775 769 769
query45 290 251 231 231
query46 1121 707 713 707
query47 2042 1971 1961 1961
query48 490 398 376 376
query49 890 357 341 341
query50 770 407 398 398
query51 6838 6787 6718 6718
query52 102 92 95 92
query53 364 281 272 272
query54 294 248 243 243
query55 91 84 86 84
query56 255 240 262 240
query57 1260 1199 1171 1171
query58 240 210 219 210
query59 2872 2567 2588 2567
query60 256 234 239 234
query61 93 89 88 88
query62 601 445 437 437
query63 300 270 272 270
query64 5272 4007 4009 4007
query65 3058 3018 3026 3018
query66 788 353 349 349
query67 15739 14755 14780 14755
query68 10252 530 531 530
query69 706 377 371 371
query70 1563 1100 1092 1092
query71 536 265 265 265
query72 7133 2526 2383 2383
query73 1692 325 321 321
query74 7099 6374 6445 6374
query75 4954 2244 2191 2191
query76 6706 930 854 854
query77 616 267 254 254
query78 11145 10166 10129 10129
query79 10809 522 538 522
query80 2182 380 371 371
query81 496 217 215 215
query82 166 82 83 82
query83 222 141 147 141
query84 282 79 77 77
query85 1217 322 303 303
query86 362 314 312 312
query87 3757 3506 3539 3506
query88 5084 2399 2389 2389
query89 479 363 357 357
query90 2356 170 171 170
query91 172 130 134 130
query92 62 45 47 45
query93 6325 486 485 485
query94 1509 172 175 172
query95 412 313 304 304
query96 616 273 269 269
query97 2619 2467 2497 2467
query98 224 215 209 209
query99 1042 824 830 824
Total cold run time: 314939 ms
Total hot run time: 182194 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026900026
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) runtime filter with probe expr should be pushed though set operator [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33010:
URL: https://github.com/apache/doris/pull/33010#issuecomment-2026876199
<details>
<summary>TPC-H: <b>Total hot run time: 38375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false
------ Round 1 ----------------------------------
q1 17600 4379 4274 4274
q2 2089 191 186 186
q3 10465 1150 1189 1150
q4 10328 754 804 754
q5 7440 2796 2690 2690
q6 217 133 131 131
q7 1011 596 597 596
q8 9210 2053 2024 2024
q9 7592 6590 6516 6516
q10 8520 3616 3537 3537
q11 468 243 238 238
q12 396 219 220 219
q13 19093 2960 2968 2960
q14 279 233 246 233
q15 531 478 459 459
q16 537 391 390 390
q17 958 558 609 558
q18 7304 6719 6637 6637
q19 1571 1424 1473 1424
q20 683 314 295 295
q21 3469 2797 2837 2797
q22 361 312 307 307
Total cold run time: 110122 ms
Total hot run time: 38375 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4265 4227 4191 4191
q2 386 258 265 258
q3 2972 2750 2703 2703
q4 1836 1561 1513 1513
q5 5286 5167 5209 5167
q6 212 122 125 122
q7 2215 1875 1909 1875
q8 3196 3290 3296 3290
q9 8512 8587 8543 8543
q10 3881 3716 3622 3622
q11 609 466 484 466
q12 757 592 578 578
q13 17658 2995 2973 2973
q14 312 274 271 271
q15 512 467 477 467
q16 486 426 435 426
q17 1783 1435 1436 1435
q18 7634 7364 7322 7322
q19 1607 1493 1549 1493
q20 1953 1775 1746 1746
q21 4987 4869 4976 4869
q22 537 477 476 476
Total cold run time: 71596 ms
Total hot run time: 53806 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org