You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "iwanttobepowerful (via GitHub)" <gi...@apache.org> on 2024/01/27 08:38:14 UTC
[PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
iwanttobepowerful opened a new pull request, #30459:
URL: https://github.com/apache/doris/pull/30459
## Proposed changes
Issue Number: close #30264
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913181950
<details>
<summary>ClickBench: <b>Total hot run time: 30.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false
query1 0.04 0.04 0.03
query2 0.05 0.02 0.02
query3 0.24 0.06 0.05
query4 1.69 0.10 0.10
query5 0.53 0.51 0.52
query6 1.18 0.63 0.63
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.56 0.51 0.50
query10 0.56 0.55 0.55
query11 0.12 0.09 0.09
query12 0.11 0.09 0.10
query13 0.59 0.60 0.61
query14 0.79 0.82 0.83
query15 0.82 0.77 0.78
query16 0.36 0.39 0.38
query17 1.08 1.05 0.98
query18 0.20 0.26 0.28
query19 1.84 1.79 1.65
query20 0.02 0.01 0.01
query21 15.35 0.59 0.56
query22 2.51 2.25 2.14
query23 17.18 0.78 0.81
query24 2.33 1.29 0.65
query25 0.29 0.12 0.19
query26 0.52 0.14 0.13
query27 0.05 0.04 0.03
query28 11.87 0.85 0.88
query29 12.56 3.16 3.10
query30 0.65 0.56 0.59
query31 2.78 0.36 0.35
query32 3.35 0.48 0.48
query33 3.23 3.23 3.23
query34 15.76 4.27 4.22
query35 4.34 4.30 4.27
query36 1.10 1.07 1.06
query37 0.06 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.01 0.02
query40 0.16 0.14 0.13
query41 0.07 0.02 0.01
query42 0.03 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.1 s
Total hot run time: 30.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913154359
<details>
<summary>ClickBench: <b>Total hot run time: 30.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.02 0.02
query3 0.23 0.05 0.06
query4 1.67 0.10 0.10
query5 0.53 0.51 0.51
query6 1.19 0.63 0.61
query7 0.02 0.02 0.01
query8 0.03 0.02 0.02
query9 0.56 0.52 0.49
query10 0.57 0.56 0.57
query11 0.12 0.09 0.09
query12 0.11 0.08 0.09
query13 0.61 0.61 0.61
query14 0.79 0.80 0.80
query15 0.79 0.79 0.79
query16 0.38 0.39 0.40
query17 0.96 1.03 1.03
query18 0.23 0.23 0.25
query19 1.83 1.80 1.79
query20 0.01 0.00 0.01
query21 15.40 0.57 0.58
query22 2.46 2.12 1.74
query23 17.64 0.89 0.70
query24 3.00 0.41 0.84
query25 0.29 0.14 0.14
query26 0.55 0.14 0.14
query27 0.05 0.06 0.05
query28 11.50 0.85 0.86
query29 12.48 3.17 3.27
query30 0.65 0.56 0.53
query31 2.78 0.36 0.35
query32 3.34 0.50 0.49
query33 3.21 3.22 3.24
query34 15.75 4.29 4.34
query35 4.47 4.35 4.41
query36 1.11 1.07 1.05
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.03 0.02 0.01
query40 0.16 0.12 0.13
query41 0.07 0.01 0.02
query42 0.03 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.82 s
Total hot run time: 30.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener merged PR #30459:
URL: https://github.com/apache/doris/pull/30459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913153256
<details>
<summary>TPC-DS: <b>Total hot run time: 174285 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false
query1 936 339 332 332
query2 6537 2070 1936 1936
query3 6921 215 203 203
query4 32816 22528 22479 22479
query5 4597 383 458 383
query6 549 175 170 170
query7 5860 292 287 287
query8 515 190 186 186
query9 10473 2304 2280 2280
query10 421 215 206 206
query11 17946 15434 15366 15366
query12 122 71 68 68
query13 1678 399 379 379
query14 9255 6540 6508 6508
query15 213 182 180 180
query16 5939 256 255 255
query17 1410 473 477 473
query18 1795 261 257 257
query19 187 141 131 131
query20 74 70 64 64
query21 206 134 143 134
query22 4775 4823 4628 4628
query23 31685 30318 30450 30318
query24 12854 2833 2805 2805
query25 596 317 307 307
query26 1979 141 135 135
query27 3569 284 286 284
query28 7629 1827 1807 1807
query29 2243 631 639 631
query30 290 134 144 134
query31 911 700 725 700
query32 76 51 50 50
query33 704 203 213 203
query34 1230 461 471 461
query35 871 754 744 744
query36 1347 1237 1187 1187
query37 177 58 62 58
query38 3297 3179 3177 3177
query39 1323 1268 1272 1268
query40 336 86 84 84
query41 38 35 34 34
query42 86 78 87 78
query43 546 470 511 470
query44 1163 684 690 684
query45 196 176 177 176
query46 1046 665 657 657
query47 1588 1527 1505 1505
query48 397 306 305 305
query49 1276 298 290 290
query50 712 312 318 312
query51 5301 5186 5134 5134
query52 90 83 77 77
query53 324 271 264 264
query54 257 190 193 190
query55 80 75 76 75
query56 173 161 161 161
query57 981 907 910 907
query58 186 166 157 157
query59 2948 2746 2742 2742
query60 211 182 174 174
query61 82 86 92 86
query62 612 345 370 345
query63 292 259 268 259
query64 5785 1797 1738 1738
query65 3312 3239 3217 3217
query66 1460 339 311 311
query67 14539 14255 14127 14127
query68 4433 480 491 480
query69 426 303 286 286
query70 1546 1476 1471 1471
query71 298 218 227 218
query72 3960 2885 2878 2878
query73 692 306 318 306
query74 6765 6282 6264 6264
query75 2873 2262 2284 2262
query76 3282 1037 992 992
query77 367 244 237 237
query78 9099 8836 8736 8736
query79 2539 505 482 482
query80 1987 327 318 318
query81 519 198 197 197
query82 742 82 86 82
query83 246 120 120 120
query84 285 75 74 74
query85 2393 343 336 336
query86 517 401 409 401
query87 3495 3247 3295 3247
query88 3881 2157 2183 2157
query89 433 352 360 352
query90 1935 188 184 184
query91 152 117 119 117
query92 54 43 42 42
query93 2433 429 407 407
query94 1397 165 162 162
query95 501 457 454 454
query96 618 311 307 307
query97 4224 4136 4137 4136
query98 205 201 190 190
query99 1058 662 657 657
Total cold run time: 291378 ms
Total hot run time: 174285 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913180482
<details>
<summary>TPC-DS: <b>Total hot run time: 173037 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false
query1 947 336 322 322
query2 6558 1947 2012 1947
query3 6694 202 195 195
query4 33558 22020 22050 22020
query5 4465 420 346 346
query6 252 151 154 151
query7 4600 255 253 253
query8 238 193 177 177
query9 8550 2265 2247 2247
query10 408 203 204 203
query11 17873 15426 15340 15340
query12 120 68 67 67
query13 1697 386 371 371
query14 9216 6438 6408 6408
query15 209 175 186 175
query16 5695 260 245 245
query17 951 472 476 472
query18 1816 261 249 249
query19 178 132 132 132
query20 70 68 73 68
query21 201 131 130 130
query22 5009 4473 4734 4473
query23 31373 30339 30379 30339
query24 7412 2765 2731 2731
query25 423 297 304 297
query26 796 135 132 132
query27 2773 276 279 276
query28 3997 1836 1810 1810
query29 895 614 610 610
query30 215 137 133 133
query31 891 702 719 702
query32 62 52 48 48
query33 418 210 210 210
query34 805 454 460 454
query35 841 758 723 723
query36 1296 1180 1180 1180
query37 94 60 57 57
query38 3289 3180 3132 3132
query39 1316 1263 1260 1260
query40 198 98 84 84
query41 37 34 34 34
query42 88 83 87 83
query43 544 496 472 472
query44 1063 677 674 674
query45 187 177 175 175
query46 1025 640 642 640
query47 1578 1556 1478 1478
query48 379 302 302 302
query49 603 280 281 280
query50 694 301 318 301
query51 5322 5217 5219 5217
query52 83 80 82 80
query53 313 263 262 262
query54 225 188 186 186
query55 79 77 70 70
query56 178 160 169 160
query57 986 905 936 905
query58 178 155 162 155
query59 2718 2714 2689 2689
query60 195 178 179 178
query61 83 86 81 81
query62 434 351 364 351
query63 293 261 267 261
query64 2541 1786 1712 1712
query65 3307 3253 3201 3201
query66 1127 310 321 310
query67 14504 14438 14338 14338
query68 3793 482 486 482
query69 446 302 293 293
query70 1504 1492 1483 1483
query71 267 207 214 207
query72 3716 2842 2831 2831
query73 684 310 306 306
query74 6610 6252 6356 6252
query75 2748 2263 2296 2263
query76 2666 1023 998 998
query77 381 231 233 231
query78 9013 8959 8571 8571
query79 2385 481 476 476
query80 2020 330 310 310
query81 495 199 200 199
query82 670 83 81 81
query83 275 119 111 111
query84 282 75 71 71
query85 1699 333 339 333
query86 518 402 372 372
query87 3425 3278 3267 3267
query88 3880 2156 2147 2147
query89 425 357 345 345
query90 2005 191 180 180
query91 150 116 119 116
query92 52 45 45 45
query93 2380 426 406 406
query94 1388 162 158 158
query95 499 461 453 453
query96 627 304 300 300
query97 4267 4121 4116 4116
query98 220 186 184 184
query99 1052 686 700 686
Total cold run time: 265705 ms
Total hot run time: 173037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913163868
<details>
<summary>TPC-DS: <b>Total hot run time: 173626 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false
query1 940 335 333 333
query2 6550 1964 1927 1927
query3 6689 201 194 194
query4 32465 22107 22113 22107
query5 4452 420 424 420
query6 252 165 173 165
query7 4609 277 274 274
query8 243 182 193 182
query9 8580 2279 2258 2258
query10 402 210 199 199
query11 18042 15436 15473 15436
query12 125 70 66 66
query13 1674 365 374 365
query14 9236 6533 6452 6452
query15 208 184 190 184
query16 5730 253 244 244
query17 954 466 480 466
query18 1807 276 254 254
query19 183 141 133 133
query20 77 77 69 69
query21 201 131 131 131
query22 4575 4695 4644 4644
query23 31306 30375 30410 30375
query24 12192 2809 2825 2809
query25 597 315 307 307
query26 1817 150 144 144
query27 3185 288 278 278
query28 7657 1827 1815 1815
query29 1865 625 594 594
query30 283 140 138 138
query31 912 716 724 716
query32 79 47 49 47
query33 696 219 205 205
query34 1181 469 463 463
query35 864 791 763 763
query36 1375 1239 1167 1167
query37 94 58 57 57
query38 3283 3176 3139 3139
query39 1317 1250 1242 1242
query40 342 97 84 84
query41 39 35 35 35
query42 88 81 83 81
query43 527 491 515 491
query44 1142 690 689 689
query45 192 178 174 174
query46 1027 659 651 651
query47 1584 1542 1493 1493
query48 381 320 311 311
query49 1170 291 292 291
query50 662 316 316 316
query51 5310 5217 5194 5194
query52 99 78 79 78
query53 316 261 263 261
query54 260 198 191 191
query55 82 80 74 74
query56 173 163 165 163
query57 998 906 889 889
query58 182 160 156 156
query59 2857 2759 2705 2705
query60 208 186 182 182
query61 83 82 82 82
query62 659 357 347 347
query63 279 259 257 257
query64 5964 1771 1740 1740
query65 3303 3249 3214 3214
query66 1242 319 306 306
query67 14589 14250 14303 14250
query68 3722 504 503 503
query69 445 296 322 296
query70 1559 1509 1535 1509
query71 287 205 214 205
query72 3756 2817 2854 2817
query73 694 327 303 303
query74 6628 6233 6193 6193
query75 2847 2256 2316 2256
query76 2600 1061 958 958
query77 378 240 239 239
query78 9141 8779 8450 8450
query79 2834 491 494 491
query80 1996 327 328 327
query81 516 199 202 199
query82 722 87 84 84
query83 230 123 119 119
query84 294 83 72 72
query85 2207 339 342 339
query86 510 405 369 369
query87 3488 3261 3248 3248
query88 3739 2156 2155 2155
query89 393 373 361 361
query90 1933 189 185 185
query91 151 120 118 118
query92 54 41 45 41
query93 2855 417 442 417
query94 1361 162 162 162
query95 509 468 460 460
query96 617 320 326 320
query97 4241 4146 4112 4112
query98 218 194 189 189
query99 1065 673 662 662
Total cold run time: 282445 ms
Total hot run time: 173626 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1914331213
<details>
<summary>TPC-DS: <b>Total hot run time: 172655 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false
query1 927 327 318 318
query2 6566 1903 1928 1903
query3 6703 198 198 198
query4 33188 22048 21995 21995
query5 4453 352 355 352
query6 247 152 149 149
query7 4611 260 258 258
query8 247 173 173 173
query9 8599 2270 2267 2267
query10 408 225 206 206
query11 17622 15390 15400 15390
query12 118 69 65 65
query13 1672 376 376 376
query14 9210 6562 6428 6428
query15 204 183 179 179
query16 5701 255 251 251
query17 948 474 488 474
query18 1804 253 254 253
query19 172 136 131 131
query20 76 64 67 64
query21 193 134 131 131
query22 4880 4747 4569 4569
query23 31228 30349 30419 30349
query24 12695 2749 2790 2749
query25 608 308 308 308
query26 1970 140 140 140
query27 3460 282 285 282
query28 7715 1818 1827 1818
query29 2308 624 618 618
query30 286 133 142 133
query31 896 710 732 710
query32 76 57 46 46
query33 702 204 205 204
query34 1300 456 471 456
query35 863 752 748 748
query36 1282 1198 1169 1169
query37 175 59 55 55
query38 3242 3143 3151 3143
query39 1313 1248 1241 1241
query40 344 87 80 80
query41 37 35 34 34
query42 85 81 83 81
query43 493 472 469 469
query44 1130 700 690 690
query45 187 177 173 173
query46 1041 647 646 646
query47 1606 1548 1520 1520
query48 374 305 294 294
query49 1226 279 286 279
query50 674 311 307 307
query51 5277 5149 5131 5131
query52 85 79 72 72
query53 318 263 270 263
query54 264 180 187 180
query55 76 77 75 75
query56 169 154 164 154
query57 983 893 915 893
query58 171 147 159 147
query59 2651 2518 2566 2518
query60 200 175 175 175
query61 86 85 84 84
query62 631 362 354 354
query63 270 260 266 260
query64 5967 1749 1736 1736
query65 3277 3223 3188 3188
query66 1465 316 310 310
query67 14330 14140 14154 14140
query68 3666 481 490 481
query69 427 296 306 296
query70 1513 1423 1477 1423
query71 288 214 206 206
query72 3770 2852 2876 2852
query73 693 307 307 307
query74 6667 6170 6123 6123
query75 2736 2311 2289 2289
query76 2682 1037 982 982
query77 434 233 225 225
query78 9031 8621 8544 8544
query79 3116 489 498 489
query80 1916 314 316 314
query81 525 195 197 195
query82 1117 78 84 78
query83 394 116 115 115
query84 284 78 70 70
query85 1767 318 321 318
query86 497 387 383 383
query87 3397 3264 3259 3259
query88 3774 2159 2148 2148
query89 404 337 374 337
query90 2091 183 185 183
query91 145 119 116 116
query92 57 44 42 42
query93 2632 414 417 414
query94 1516 158 158 158
query95 501 465 449 449
query96 634 314 314 314
query97 4288 4133 4163 4133
query98 221 193 189 189
query99 1132 671 688 671
Total cold run time: 284400 ms
Total hot run time: 172655 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913091800
<details>
<summary>TPC-H: <b>Total hot run time: 37530 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false
------ Round 1 ----------------------------------
q1 17708 4792 5081 4792
q2 2046 144 135 135
q3 10705 925 938 925
q4 4746 753 738 738
q5 7678 2901 2856 2856
q6 184 119 119 119
q7 1095 746 706 706
q8 9370 2028 1981 1981
q9 7222 6313 6333 6313
q10 8069 2459 2401 2401
q11 420 196 196 196
q12 767 278 273 273
q13 18027 3286 3291 3286
q14 274 239 246 239
q15 524 486 492 486
q16 464 396 392 392
q17 924 508 568 508
q18 6870 6153 6175 6153
q19 1568 1367 1378 1367
q20 621 348 333 333
q21 6796 3061 3025 3025
q22 815 308 306 306
Total cold run time: 106893 ms
Total hot run time: 37530 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5121 5031 5063 5031
q2 326 229 236 229
q3 2985 2630 2617 2617
q4 1817 1482 1502 1482
q5 5314 5182 5341 5182
q6 196 117 113 113
q7 2153 1764 1823 1764
q8 3118 3152 3260 3152
q9 8684 8623 9555 8623
q10 5656 3451 3475 3451
q11 563 466 484 466
q12 750 583 597 583
q13 16949 3096 3117 3096
q14 273 262 272 262
q15 528 493 487 487
q16 505 472 464 464
q17 1867 1775 1708 1708
q18 8719 16202 8620 8620
q19 26820 1544 1538 1538
q20 5198 1899 1893 1893
q21 14349 4918 4834 4834
q22 1021 487 448 448
Total cold run time: 112912 ms
Total hot run time: 56043 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1914340763
<details>
<summary>ClickBench: <b>Total hot run time: 30.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false
query1 0.04 0.02 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.69 0.10 0.09
query5 0.53 0.52 0.52
query6 1.27 0.64 0.62
query7 0.02 0.01 0.01
query8 0.04 0.02 0.02
query9 0.55 0.50 0.48
query10 0.54 0.54 0.54
query11 0.11 0.09 0.08
query12 0.11 0.08 0.09
query13 0.60 0.60 0.61
query14 0.78 0.81 0.80
query15 0.79 0.78 0.78
query16 0.38 0.39 0.38
query17 1.02 1.04 1.04
query18 0.22 0.24 0.24
query19 1.85 1.70 1.79
query20 0.02 0.01 0.01
query21 15.40 0.58 0.57
query22 2.54 2.46 2.10
query23 17.38 0.76 0.80
query24 2.84 0.88 0.45
query25 0.34 0.25 0.05
query26 0.24 0.13 0.14
query27 0.05 0.05 0.04
query28 12.36 0.87 0.83
query29 12.56 3.06 3.10
query30 0.65 0.53 0.56
query31 2.81 0.35 0.35
query32 3.35 0.47 0.48
query33 3.21 3.21 3.20
query34 15.66 4.33 4.38
query35 4.29 4.28 4.37
query36 1.12 1.07 1.07
query37 0.07 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.12 0.16
query41 0.08 0.01 0.02
query42 0.02 0.02 0.02
query43 0.03 0.01 0.02
Total cold run time: 106.06 s
Total hot run time: 30.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469140415
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
try replace
```
if (conjuncts.stream().flatMap(c -> c.children().stream())
.anyMatch(MarkJoinSlotReference.class::isInstance)) {
return new LogicalProject<>(applyPlan.getOutput().stream()
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
return newFilter;
}
```
with
```
return new LogicalProject<>(filter.getOutput().stream()
.collect(ImmutableList.toImmutableList()), newFilter);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913178073
<details>
<summary>TPC-H: <b>Total hot run time: 37235 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false
------ Round 1 ----------------------------------
q1 17729 4773 4994 4773
q2 2059 154 136 136
q3 10686 925 912 912
q4 4727 728 684 684
q5 7699 2818 2758 2758
q6 184 120 120 120
q7 1150 712 702 702
q8 9376 2016 2008 2008
q9 7266 6346 6293 6293
q10 8105 2384 2432 2384
q11 417 198 200 198
q12 749 286 291 286
q13 17998 3305 3272 3272
q14 279 245 249 245
q15 525 498 495 495
q16 456 419 415 415
q17 926 550 495 495
q18 6978 5996 5987 5987
q19 2223 1388 1369 1369
q20 625 345 337 337
q21 6801 3103 3082 3082
q22 789 284 301 284
Total cold run time: 107747 ms
Total hot run time: 37235 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5210 4982 5015 4982
q2 352 244 243 243
q3 3022 2711 2659 2659
q4 1837 1581 1534 1534
q5 5216 5222 5265 5222
q6 200 117 115 115
q7 2148 1728 1759 1728
q8 3098 3250 3262 3250
q9 8644 8913 18997 8913
q10 5638 3392 3545 3392
q11 566 469 473 469
q12 759 614 613 613
q13 16961 3073 3057 3057
q14 277 251 256 251
q15 538 507 484 484
q16 513 472 446 446
q17 1952 1771 1762 1762
q18 8861 8506 11537 8506
q19 23981 1585 1551 1551
q20 4567 1933 1881 1881
q21 14780 4900 4767 4767
q22 1048 524 465 465
Total cold run time: 110168 ms
Total hot run time: 56290 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469044186
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
with this change
```
mysql> explain parsed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| UnboundResultSink[11] ( ) |
| +--LogicalCTE ( aliasQueries=[LogicalSubQueryAlias ( qualifier=[cte1] )] ) |
| |-*LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[8] ( distinct=false, projects=[*], excepts=[] ) |
| | +--LogicalCheckPolicy ( ) |
| | +--UnboundRelation ( id=RelationId#2, nameParts=t123 ) |
| +--LogicalHaving ( predicates=('tm > (SCALARSUBQUERY) SubqueryExpr ( QueryPlan=LogicalProject[4] ( distinct=false, projects=[UnboundAlias('Apple')], excepts=[] ), CorrelatedSlots=[], typeCoercionExpr=null )) ) |
| +--LogicalAggregate[2] ( groupByExpr=['tm], outputExpr=['tm], hasRepeat=false ) |
| +--LogicalCheckPolicy ( ) |
| +--UnboundRelation ( id=RelationId#0, nameParts=cte1 ) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
10 rows in set (0.00 sec)
mysql> explain analyzed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[43] ( outputExprs=[tm#1] ) |
| +--LogicalCteAnchor[41] ( cteId=CTEId#0 ) |
| |--LogicalCteProducer[18] ( cteId=CTEId#0 ) |
| | +--LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[16] ( distinct=false, projects=[tm#0], excepts=[] ) |
| | +--LogicalOlapScan ( qualified=demo.t123, indexName=<index_not_selected>, selectedIndexId=10097, preAgg=ON ) |
| +--LogicalProject[40] ( distinct=false, projects=[tm#1], excepts=[] ) |
| +--LogicalFilter[39] ( predicates=(tm#1 > 'Apple'#3) ) |
| +--LogicalProject[38] ( distinct=false, projects=[tm#1, 'Apple'#3], excepts=[] ) |
| +--LogicalApply ( correlationSlot=[], correlationFilter=Optional.empty, isMarkJoin=false, isMarkJoinSlotNotNull=false, MarkJoinSlotReference=empty ) |
| |--LogicalProject[34] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalAggregate[33] ( groupByExpr=[tm#1], outputExpr=[tm#1], hasRepeat=false ) |
| | +--LogicalProject[32] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalCteConsumer[19] ( cteId=CTEId#0, relationId=RelationId#0, name=cte1 ) |
| +--LogicalProject[27] ( distinct=false, projects=['Apple' AS `'Apple'`#3], excepts=[] ) |
| +--LogicalOneRowRelation ( projects=['Apple' AS `'Apple'`#2] ) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
16 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469044186
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
with this pr
```
mysql> explain parsed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| UnboundResultSink[11] ( ) |
| +--LogicalCTE ( aliasQueries=[LogicalSubQueryAlias ( qualifier=[cte1] )] ) |
| |-*LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[8] ( distinct=false, projects=[*], excepts=[] ) |
| | +--LogicalCheckPolicy ( ) |
| | +--UnboundRelation ( id=RelationId#2, nameParts=t123 ) |
| +--LogicalHaving ( predicates=('tm > (SCALARSUBQUERY) SubqueryExpr ( QueryPlan=LogicalProject[4] ( distinct=false, projects=[UnboundAlias('Apple')], excepts=[] ), CorrelatedSlots=[], typeCoercionExpr=null )) ) |
| +--LogicalAggregate[2] ( groupByExpr=['tm], outputExpr=['tm], hasRepeat=false ) |
| +--LogicalCheckPolicy ( ) |
| +--UnboundRelation ( id=RelationId#0, nameParts=cte1 ) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
10 rows in set (0.00 sec)
mysql> explain analyzed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[43] ( outputExprs=[tm#1] ) |
| +--LogicalCteAnchor[41] ( cteId=CTEId#0 ) |
| |--LogicalCteProducer[18] ( cteId=CTEId#0 ) |
| | +--LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[16] ( distinct=false, projects=[tm#0], excepts=[] ) |
| | +--LogicalOlapScan ( qualified=demo.t123, indexName=<index_not_selected>, selectedIndexId=10097, preAgg=ON ) |
| +--LogicalProject[40] ( distinct=false, projects=[tm#1], excepts=[] ) |
| +--LogicalFilter[39] ( predicates=(tm#1 > 'Apple'#3) ) |
| +--LogicalProject[38] ( distinct=false, projects=[tm#1, 'Apple'#3], excepts=[] ) |
| +--LogicalApply ( correlationSlot=[], correlationFilter=Optional.empty, isMarkJoin=false, isMarkJoinSlotNotNull=false, MarkJoinSlotReference=empty ) |
| |--LogicalProject[34] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalAggregate[33] ( groupByExpr=[tm#1], outputExpr=[tm#1], hasRepeat=false ) |
| | +--LogicalProject[32] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalCteConsumer[19] ( cteId=CTEId#0, relationId=RelationId#0, name=cte1 ) |
| +--LogicalProject[27] ( distinct=false, projects=['Apple' AS `'Apple'`#3], excepts=[] ) |
| +--LogicalOneRowRelation ( projects=['Apple' AS `'Apple'`#2] ) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
16 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1914346095
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.1 seconds inserted 10000000 Rows, about 709K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1915936167
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.3 seconds inserted 10000000 Rows, about 751K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913097357
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1915928404
<details>
<summary>TPC-DS: <b>Total hot run time: 172409 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false
query1 934 344 325 325
query2 6565 1894 1821 1821
query3 6696 201 191 191
query4 27335 21987 22140 21987
query5 4459 355 343 343
query6 260 162 169 162
query7 4604 262 253 253
query8 253 172 190 172
query9 9145 2278 2253 2253
query10 416 224 196 196
query11 16859 15373 15497 15373
query12 128 66 64 64
query13 1682 380 371 371
query14 9380 6477 6424 6424
query15 225 183 188 183
query16 6472 255 247 247
query17 1902 462 478 462
query18 1909 251 245 245
query19 178 134 129 129
query20 70 73 68 68
query21 193 123 130 123
query22 4995 4734 4758 4734
query23 31070 30268 30178 30178
query24 8814 2686 2762 2686
query25 538 313 312 312
query26 826 137 137 137
query27 2748 281 295 281
query28 5898 1826 1824 1824
query29 928 622 610 610
query30 277 138 139 138
query31 917 706 700 700
query32 88 55 52 52
query33 501 203 202 202
query34 825 435 453 435
query35 860 756 759 756
query36 1229 1186 1138 1138
query37 93 55 62 55
query38 3260 3137 3123 3123
query39 1313 1253 1250 1250
query40 189 87 84 84
query41 37 35 34 34
query42 83 78 80 78
query43 510 481 491 481
query44 1011 682 687 682
query45 191 180 176 176
query46 1026 641 628 628
query47 1608 1528 1529 1528
query48 380 314 297 297
query49 1117 285 278 278
query50 681 302 308 302
query51 5320 5112 5136 5112
query52 95 83 82 82
query53 322 255 257 255
query54 230 194 179 179
query55 82 75 80 75
query56 170 162 160 160
query57 995 921 909 909
query58 173 152 158 152
query59 2632 2446 2299 2299
query60 194 174 173 173
query61 86 87 86 86
query62 571 368 347 347
query63 277 255 261 255
query64 4042 1793 1740 1740
query65 3301 3234 3213 3213
query66 1136 326 326 326
query67 14438 14354 14096 14096
query68 4037 499 509 499
query69 441 311 312 311
query70 1451 1517 1489 1489
query71 280 212 209 209
query72 3729 2867 2889 2867
query73 686 309 309 309
query74 6578 6307 6300 6300
query75 2837 2343 2267 2267
query76 2975 972 927 927
query77 378 233 225 225
query78 9172 8772 8558 8558
query79 2658 493 498 493
query80 2075 334 333 333
query81 528 196 197 196
query82 719 84 81 81
query83 251 119 115 115
query84 277 77 72 72
query85 1947 356 323 323
query86 506 380 385 380
query87 3439 3274 3324 3274
query88 3912 2179 2170 2170
query89 413 364 338 338
query90 1905 187 185 185
query91 153 117 118 117
query92 51 46 44 44
query93 2507 422 424 422
query94 1260 157 161 157
query95 508 467 449 449
query96 624 334 315 315
query97 4237 4115 4124 4115
query98 202 197 189 189
query99 1005 694 650 650
Total cold run time: 267483 ms
Total hot run time: 172409 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469044186
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
```
mysql> explain parsed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| UnboundResultSink[11] ( ) |
| +--LogicalCTE ( aliasQueries=[LogicalSubQueryAlias ( qualifier=[cte1] )] ) |
| |-*LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[8] ( distinct=false, projects=[*], excepts=[] ) |
| | +--LogicalCheckPolicy ( ) |
| | +--UnboundRelation ( id=RelationId#2, nameParts=t123 ) |
| +--LogicalHaving ( predicates=('tm > (SCALARSUBQUERY) SubqueryExpr ( QueryPlan=LogicalProject[4] ( distinct=false, projects=[UnboundAlias('Apple')], excepts=[] ), CorrelatedSlots=[], typeCoercionExpr=null )) ) |
| +--LogicalAggregate[2] ( groupByExpr=['tm], outputExpr=['tm], hasRepeat=false ) |
| +--LogicalCheckPolicy ( ) |
| +--UnboundRelation ( id=RelationId#0, nameParts=cte1 ) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
10 rows in set (0.00 sec)
mysql> explain analyzed plan with cte1 as(select * from t123) select tm from cte1 group by tm having tm > (select 'Apple');
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalResultSink[43] ( outputExprs=[tm#1] ) |
| +--LogicalCteAnchor[41] ( cteId=CTEId#0 ) |
| |--LogicalCteProducer[18] ( cteId=CTEId#0 ) |
| | +--LogicalSubQueryAlias ( qualifier=[cte1] ) |
| | +--LogicalProject[16] ( distinct=false, projects=[tm#0], excepts=[] ) |
| | +--LogicalOlapScan ( qualified=demo.t123, indexName=<index_not_selected>, selectedIndexId=10097, preAgg=ON ) |
| +--LogicalProject[40] ( distinct=false, projects=[tm#1], excepts=[] ) |
| +--LogicalFilter[39] ( predicates=(tm#1 > 'Apple'#3) ) |
| +--LogicalProject[38] ( distinct=false, projects=[tm#1, 'Apple'#3], excepts=[] ) |
| +--LogicalApply ( correlationSlot=[], correlationFilter=Optional.empty, isMarkJoin=false, isMarkJoinSlotNotNull=false, MarkJoinSlotReference=empty ) |
| |--LogicalProject[34] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalAggregate[33] ( groupByExpr=[tm#1], outputExpr=[tm#1], hasRepeat=false ) |
| | +--LogicalProject[32] ( distinct=false, projects=[tm#1], excepts=[] ) |
| | +--LogicalCteConsumer[19] ( cteId=CTEId#0, relationId=RelationId#0, name=cte1 ) |
| +--LogicalProject[27] ( distinct=false, projects=['Apple' AS `'Apple'`#3], excepts=[] ) |
| +--LogicalOneRowRelation ( projects=['Apple' AS `'Apple'`#2] ) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
16 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1915895564
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1916513626
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913096425
<details>
<summary>ClickBench: <b>Total hot run time: 31.9 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false
query1 0.04 0.04 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.68 0.10 0.10
query5 0.52 0.51 0.52
query6 1.20 0.64 0.66
query7 0.02 0.01 0.01
query8 0.03 0.02 0.03
query9 0.55 0.49 0.51
query10 0.55 0.57 0.56
query11 0.12 0.08 0.09
query12 0.11 0.08 0.09
query13 0.61 0.61 0.62
query14 0.79 0.79 0.80
query15 0.80 0.78 0.75
query16 0.39 0.39 0.38
query17 1.02 1.00 0.96
query18 0.20 0.25 0.23
query19 1.94 1.77 1.79
query20 0.02 0.01 0.01
query21 15.42 0.57 0.58
query22 2.62 2.57 2.80
query23 17.35 0.76 0.75
query24 2.25 1.28 1.10
query25 0.41 0.29 0.15
query26 0.47 0.14 0.14
query27 0.05 0.05 0.05
query28 11.64 0.86 0.81
query29 12.50 3.17 3.14
query30 0.69 0.53 0.56
query31 2.79 0.35 0.35
query32 3.37 0.48 0.48
query33 3.23 3.24 3.23
query34 15.77 4.43 4.36
query35 4.33 4.32 4.25
query36 1.12 1.06 1.07
query37 0.07 0.05 0.05
query38 0.05 0.03 0.02
query39 0.03 0.01 0.02
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.32 s
Total hot run time: 31.9 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469193274
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
@starocean999 you're right! Thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913094863
<details>
<summary>TPC-DS: <b>Total hot run time: 172995 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false
query1 938 335 328 328
query2 6568 2057 1867 1867
query3 6709 197 195 195
query4 32414 22187 22078 22078
query5 4457 357 351 351
query6 254 166 156 156
query7 4610 270 264 264
query8 241 182 183 182
query9 8634 2293 2286 2286
query10 407 212 199 199
query11 16441 15335 15284 15284
query12 122 71 66 66
query13 1705 385 388 385
query14 9196 6333 6493 6333
query15 211 183 183 183
query16 5714 254 250 250
query17 951 487 467 467
query18 1800 259 251 251
query19 190 133 131 131
query20 70 74 65 65
query21 197 126 135 126
query22 4910 4720 4573 4573
query23 31222 30470 30419 30419
query24 12476 2798 2790 2790
query25 599 301 313 301
query26 1951 144 140 140
query27 3177 287 283 283
query28 7658 1817 1821 1817
query29 2259 622 617 617
query30 284 134 147 134
query31 900 719 718 718
query32 78 52 51 51
query33 692 215 203 203
query34 1192 458 471 458
query35 859 763 760 760
query36 1329 1256 1217 1217
query37 95 59 58 58
query38 3244 3190 3149 3149
query39 1317 1259 1259 1259
query40 341 92 85 85
query41 41 35 36 35
query42 93 82 80 80
query43 530 470 493 470
query44 1126 682 687 682
query45 196 181 175 175
query46 1068 668 659 659
query47 1609 1527 1526 1526
query48 385 303 311 303
query49 1218 287 286 286
query50 677 313 298 298
query51 5384 5145 5198 5145
query52 89 88 77 77
query53 323 259 273 259
query54 246 193 186 186
query55 82 74 71 71
query56 180 159 158 158
query57 1007 891 909 891
query58 185 152 162 152
query59 2744 2594 2425 2425
query60 207 182 178 178
query61 93 85 83 83
query62 617 374 359 359
query63 278 269 260 260
query64 5976 1797 1748 1748
query65 3320 3249 3225 3225
query66 1462 329 322 322
query67 14416 14138 14153 14138
query68 3768 485 520 485
query69 436 313 298 298
query70 1529 1484 1507 1484
query71 268 226 214 214
query72 3746 2850 2834 2834
query73 702 310 310 310
query74 6640 6278 6199 6199
query75 2840 2287 2280 2280
query76 2588 1032 1102 1032
query77 327 234 224 224
query78 9134 8946 8505 8505
query79 2804 484 474 474
query80 2009 327 314 314
query81 517 200 206 200
query82 695 90 81 81
query83 244 126 120 120
query84 289 77 73 73
query85 2244 343 336 336
query86 518 380 414 380
query87 3436 3279 3288 3279
query88 3777 2175 2168 2168
query89 422 350 364 350
query90 1921 191 184 184
query91 146 118 117 117
query92 58 43 44 43
query93 2680 437 440 437
query94 1350 161 163 161
query95 510 462 454 454
query96 634 314 314 314
query97 4254 4119 4109 4109
query98 213 195 184 184
query99 1060 704 682 682
Total cold run time: 281723 ms
Total hot run time: 172995 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913166081
<details>
<summary>ClickBench: <b>Total hot run time: 30.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.03
query3 0.23 0.06 0.05
query4 1.65 0.10 0.10
query5 0.53 0.51 0.51
query6 1.19 0.63 0.61
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.53 0.51 0.50
query10 0.55 0.57 0.57
query11 0.11 0.08 0.09
query12 0.11 0.09 0.10
query13 0.62 0.60 0.60
query14 0.80 0.80 0.80
query15 0.79 0.76 0.77
query16 0.39 0.39 0.40
query17 1.02 0.99 1.03
query18 0.22 0.25 0.25
query19 1.86 1.70 1.74
query20 0.01 0.02 0.02
query21 15.40 0.58 0.57
query22 2.27 2.52 1.52
query23 17.38 1.08 0.78
query24 2.51 0.56 1.43
query25 0.29 0.18 0.17
query26 0.53 0.14 0.13
query27 0.04 0.04 0.04
query28 10.86 0.86 0.86
query29 12.59 3.22 3.25
query30 0.67 0.56 0.53
query31 2.78 0.35 0.36
query32 3.34 0.48 0.48
query33 3.24 3.19 3.25
query34 15.75 4.37 4.36
query35 4.31 4.29 4.29
query36 1.11 1.06 1.07
query37 0.07 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.17 0.13 0.13
query41 0.07 0.02 0.01
query42 0.03 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.26 s
Total hot run time: 30.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469035697
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -410,6 +417,9 @@ private LogicalPlan addApply(SubqueryExpr subquery, LogicalPlan childPlan,
? ImmutableList.of(subquery.getQueryPlan().getOutput().get(0)) : ImmutableList.of())
.build();
+ if (subqueryToMarkJoinSlot.get(subquery).isPresent() || needAddScalarSubqueryOutputToProjects) {
Review Comment:
if using filter's output to create a new top project, this is not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469050270
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
use old filter's output `LogicalFilter[39] ( predicates=(tm#1 > 'Apple'#3) )` to create a new project, will have 2 cols: `tm#1` and `'Apple'#3`
@starocean999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913079378
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1914276838
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1915933386
<details>
<summary>ClickBench: <b>Total hot run time: 31.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.02 0.02
query3 0.23 0.07 0.06
query4 1.66 0.10 0.10
query5 0.53 0.52 0.51
query6 1.21 0.64 0.64
query7 0.01 0.01 0.00
query8 0.04 0.02 0.02
query9 0.57 0.51 0.50
query10 0.57 0.56 0.55
query11 0.12 0.08 0.09
query12 0.11 0.09 0.09
query13 0.61 0.61 0.61
query14 0.80 0.80 0.80
query15 0.81 0.79 0.78
query16 0.37 0.38 0.39
query17 1.04 1.01 1.04
query18 0.22 0.28 0.22
query19 1.83 1.70 1.72
query20 0.01 0.01 0.01
query21 15.41 0.59 0.58
query22 2.71 2.54 2.00
query23 17.42 0.86 0.86
query24 2.31 0.92 0.73
query25 0.29 0.12 0.14
query26 0.48 0.15 0.14
query27 0.05 0.05 0.06
query28 12.19 0.84 0.83
query29 12.57 3.20 3.21
query30 0.57 0.50 0.50
query31 2.79 0.34 0.35
query32 3.36 0.49 0.48
query33 3.25 3.20 3.22
query34 15.82 4.34 4.32
query35 4.35 4.28 4.33
query36 1.14 1.07 1.06
query37 0.07 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.14 0.13
query41 0.07 0.02 0.02
query42 0.03 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.96 s
Total hot run time: 31.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1915918282
<details>
<summary>TPC-H: <b>Total hot run time: 36532 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false
------ Round 1 ----------------------------------
q1 17684 4443 4418 4418
q2 2039 139 127 127
q3 10601 897 879 879
q4 4649 753 702 702
q5 7663 2693 2847 2693
q6 183 119 119 119
q7 1113 729 707 707
q8 9270 1954 1982 1954
q9 7201 6313 6290 6290
q10 8069 2404 2450 2404
q11 415 201 192 192
q12 743 279 278 278
q13 18019 3283 3313 3283
q14 278 246 253 246
q15 529 488 482 482
q16 463 415 392 392
q17 931 509 506 506
q18 6828 5900 5880 5880
q19 1561 1456 1355 1355
q20 579 309 330 309
q21 6962 3009 3032 3009
q22 806 311 307 307
Total cold run time: 106586 ms
Total hot run time: 36532 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4459 4429 4370 4370
q2 332 237 241 237
q3 2994 2838 2819 2819
q4 1865 1667 1677 1667
q5 5180 5266 5234 5234
q6 188 112 115 112
q7 2181 1786 1795 1786
q8 3104 3243 3241 3241
q9 8399 8210 8213 8210
q10 5801 3600 3522 3522
q11 550 474 455 455
q12 728 591 572 572
q13 13573 3076 3080 3076
q14 281 264 259 259
q15 532 495 489 489
q16 522 474 467 467
q17 1829 1714 1672 1672
q18 7987 7704 7498 7498
q19 8805 1554 1543 1543
q20 2153 1909 1901 1901
q21 7143 4735 4772 4735
q22 513 471 483 471
Total cold run time: 79119 ms
Total hot run time: 54336 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1914311472
<details>
<summary>TPC-H: <b>Total hot run time: 37710 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false
------ Round 1 ----------------------------------
q1 17817 4959 5104 4959
q2 2806 146 143 143
q3 11530 942 931 931
q4 5108 827 705 705
q5 7877 2944 2769 2769
q6 190 124 124 124
q7 1195 742 738 738
q8 9486 1992 2033 1992
q9 7232 6339 6338 6338
q10 8049 2441 2416 2416
q11 410 204 189 189
q12 755 278 279 278
q13 18015 3348 3298 3298
q14 270 248 251 248
q15 529 492 491 491
q16 462 398 398 398
q17 949 579 496 496
q18 6953 6180 6227 6180
q19 1573 1349 1425 1349
q20 584 359 343 343
q21 6630 3171 3029 3029
q22 810 296 301 296
Total cold run time: 109230 ms
Total hot run time: 37710 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5166 4849 4850 4849
q2 342 225 236 225
q3 3011 2773 2742 2742
q4 1843 1603 1570 1570
q5 5326 5315 5235 5235
q6 212 116 115 115
q7 2188 1826 1823 1823
q8 3142 3196 3237 3196
q9 8575 8535 14087 8535
q10 5477 3488 3473 3473
q11 546 474 465 465
q12 770 604 605 604
q13 16937 3059 3050 3050
q14 277 256 268 256
q15 532 501 493 493
q16 509 455 469 455
q17 2059 1651 1716 1651
q18 8825 11628 8792 8792
q19 25184 1545 1493 1493
q20 4593 1908 1880 1880
q21 16248 4746 4794 4746
q22 1603 461 496 461
Total cold run time: 113365 ms
Total hot run time: 56109 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913182690
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913166745
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913142602
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1916513688
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469035570
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
I think the better way to do this is using old filter's output to create a new project node on top of current result root plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on code in PR #30459:
URL: https://github.com/apache/doris/pull/30459#discussion_r1469050270
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java:
##########
@@ -140,6 +140,13 @@ public List<Rule> buildRules() {
.filter(s -> !(s instanceof MarkJoinSlotReference))
.collect(ImmutableList.toImmutableList()), newFilter);
} else {
+ if (applyPlan instanceof LogicalProject) {
Review Comment:
use old filter's output `LogicalFilter[39] ( predicates=(tm#1 > 'Apple'#3) )` to create a new project, will have 2 cols: tm#1 and 'Apple'#3
@starocean999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913154998
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913150370
<details>
<summary>TPC-H: <b>Total hot run time: 37524 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false
------ Round 1 ----------------------------------
q1 17679 5801 4857 4857
q2 2037 140 130 130
q3 10712 923 940 923
q4 4698 731 745 731
q5 7698 2956 2905 2905
q6 186 118 117 117
q7 1143 722 705 705
q8 9438 2038 2005 2005
q9 7210 6344 6271 6271
q10 8099 2434 2417 2417
q11 409 198 208 198
q12 795 264 265 264
q13 18001 3310 3310 3310
q14 269 242 239 239
q15 525 493 485 485
q16 454 424 437 424
q17 921 453 489 453
q18 6957 6162 6111 6111
q19 4952 1389 1363 1363
q20 649 329 342 329
q21 6648 3083 2980 2980
q22 813 321 307 307
Total cold run time: 110293 ms
Total hot run time: 37524 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4999 5113 4942 4942
q2 359 230 231 230
q3 3012 2688 2701 2688
q4 1851 1558 1565 1558
q5 5355 5321 5310 5310
q6 203 116 113 113
q7 2135 1798 1780 1780
q8 3118 3182 3266 3182
q9 8664 8686 14759 8686
q10 5507 3464 3548 3464
q11 567 483 480 480
q12 721 592 623 592
q13 16926 3135 3049 3049
q14 281 249 259 249
q15 537 492 492 492
q16 531 474 495 474
q17 1947 1787 1696 1696
q18 9115 9958 8504 8504
q19 24130 1547 1539 1539
q20 4615 1906 1899 1899
q21 16053 5014 4889 4889
q22 1043 499 466 466
Total cold run time: 111669 ms
Total hot run time: 56282 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913155204
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "iwanttobepowerful (via GitHub)" <gi...@apache.org>.
iwanttobepowerful commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913173417
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](Nereids) add logical project to prevent extra wrong column [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30459:
URL: https://github.com/apache/doris/pull/30459#issuecomment-1913159298
<details>
<summary>TPC-H: <b>Total hot run time: 38016 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false
------ Round 1 ----------------------------------
q1 18015 5162 5172 5162
q2 2706 143 136 136
q3 11235 957 931 931
q4 4923 727 733 727
q5 8497 2920 2827 2827
q6 194 122 122 122
q7 1168 759 746 746
q8 9425 2008 2023 2008
q9 7219 6339 6335 6335
q10 8125 2379 2425 2379
q11 420 207 214 207
q12 796 285 284 284
q13 18018 3296 3286 3286
q14 280 240 249 240
q15 528 494 475 475
q16 479 422 403 403
q17 939 559 520 520
q18 7069 6070 6130 6070
q19 1579 1384 1432 1384
q20 586 349 361 349
q21 7155 3145 3125 3125
q22 812 304 300 300
Total cold run time: 110168 ms
Total hot run time: 38016 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5028 4910 4922 4910
q2 360 245 236 236
q3 3027 2848 2772 2772
q4 1802 1601 1491 1491
q5 5323 5361 5131 5131
q6 203 120 115 115
q7 2171 1843 1802 1802
q8 3115 3157 3235 3157
q9 8668 8617 11278 8617
q10 5683 3409 3449 3409
q11 544 482 481 481
q12 757 602 656 602
q13 16952 3164 3101 3101
q14 273 269 259 259
q15 541 490 496 490
q16 514 463 463 463
q17 2030 1638 1728 1638
q18 9117 11344 8874 8874
q19 23910 1516 1502 1502
q20 4589 1909 1895 1895
q21 15727 4828 4784 4784
q22 1211 478 488 478
Total cold run time: 111545 ms
Total hot run time: 56207 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org