You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2024/01/07 13:24:55 UTC
[PR] [rf](nereids)prune rf for external db according to jump count [doris]
englefly opened a new pull request, #29634:
URL: https://github.com/apache/doris/pull/29634
## Proposed changes
tpch 5/8
prune rfs whoes jump > 2
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880559981
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880485916
<details>
<summary>TPC-DS: <b>Total hot run time: 183822 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e86134feb68ab70496682f782f64226a126049fc, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 959 341 338 338
query2 6558 2049 1965 1965
query3 6736 223 225 223
query4 25592 22402 22610 22402
query5 6081 605 552 552
query6 333 216 200 200
query7 4641 311 298 298
query8 255 235 206 206
query9 8668 2989 2997 2989
query10 572 331 326 326
query11 16163 15597 15667 15597
query12 170 97 96 96
query13 1676 356 343 343
query14 13558 7565 7592 7565
query15 285 245 215 215
query16 6386 338 329 329
query17 1723 520 511 511
query18 1973 304 301 301
query19 277 164 160 160
query20 108 93 107 93
query21 186 93 94 93
query22 5083 5028 4889 4889
query23 31759 31390 31303 31303
query24 11912 2956 2877 2877
query25 617 359 356 356
query26 1795 172 171 171
query27 2847 308 306 306
query28 6843 2187 2175 2175
query29 2024 428 407 407
query30 288 144 149 144
query31 1004 815 814 814
query32 145 87 72 72
query33 843 344 343 343
query34 870 469 474 469
query35 954 854 852 852
query36 1344 1191 1234 1191
query37 203 79 82 79
query38 3495 3351 3357 3351
query39 1354 1292 1290 1290
query40 302 88 90 88
query41 41 39 38 38
query42 106 104 110 104
query43 573 515 536 515
query44 1175 822 807 807
query45 219 208 197 197
query46 1071 687 682 682
query47 1684 1613 1702 1613
query48 390 285 300 285
query49 1210 327 310 310
query50 725 324 334 324
query51 5543 5283 5285 5283
query52 119 101 98 98
query53 227 172 164 164
query54 1464 699 713 699
query55 115 100 101 100
query56 253 255 269 255
query57 1073 1020 976 976
query58 306 262 278 262
query59 3219 2933 2883 2883
query60 335 301 297 297
query61 144 140 132 132
query62 587 473 482 473
query63 186 178 170 170
query64 5930 1704 1655 1655
query65 3396 3351 3336 3336
query66 1331 375 366 366
query67 15680 15290 15610 15290
query68 13826 539 563 539
query69 607 322 334 322
query70 1839 1545 1576 1545
query71 592 317 291 291
query72 5314 3435 3435 3435
query73 3473 340 329 329
query74 7003 6496 6588 6496
query75 5478 2298 2290 2290
query76 6327 992 1126 992
query77 774 313 350 313
query78 9073 8591 8538 8538
query79 3084 525 523 523
query80 579 333 347 333
query81 462 213 215 213
query82 247 106 101 101
query83 188 146 153 146
query84 246 56 57 56
query85 942 271 262 262
query86 439 397 444 397
query87 3634 3500 3455 3455
query88 3060 2493 2530 2493
query89 376 299 291 291
query90 1926 250 252 250
query91 121 91 93 91
query92 81 70 80 70
query93 1299 501 485 485
query94 927 221 249 221
query95 567 504 484 484
query96 639 345 335 335
query97 4436 4210 4226 4210
query98 241 224 214 214
query99 1141 809 851 809
Total cold run time: 302625 ms
Total hot run time: 183822 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880476044
<details>
<summary>TPC-H: <b>Total hot run time: 38201 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e86134feb68ab70496682f782f64226a126049fc, data reload: false
------ Round 1 ----------------------------------
q1 17675 4951 4947 4947
q2 2020 156 141 141
q3 10589 1141 1125 1125
q4 10481 765 786 765
q5 7854 2862 2861 2861
q6 201 121 120 120
q7 901 509 497 497
q8 9286 1966 2010 1966
q9 6461 6388 6324 6324
q10 8235 3068 3010 3010
q11 443 208 200 200
q12 382 223 219 219
q13 18122 3370 3373 3370
q14 242 215 200 200
q15 544 501 505 501
q16 446 394 400 394
q17 957 695 518 518
q18 7308 6596 6657 6596
q19 1537 1505 1503 1503
q20 716 311 326 311
q21 2711 2419 2336 2336
q22 350 297 301 297
Total cold run time: 107461 ms
Total hot run time: 38201 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4909 4893 4922 4893
q2 339 227 244 227
q3 3272 3311 3270 3270
q4 2118 2064 2023 2023
q5 5802 5789 5753 5753
q6 198 115 115 115
q7 2305 1912 1862 1862
q8 3375 3417 3462 3417
q9 8722 8695 8699 8695
q10 3771 3838 3847 3838
q11 545 440 455 440
q12 785 643 629 629
q13 7596 3186 3161 3161
q14 279 286 261 261
q15 544 503 503 503
q16 543 498 504 498
q17 1862 1830 1855 1830
q18 8614 8356 8264 8264
q19 1580 1615 1617 1615
q20 2169 1934 1957 1934
q21 5613 5299 5207 5207
q22 502 476 446 446
Total cold run time: 65443 ms
Total hot run time: 58881 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880067622
<details>
<summary>TPC-H: <b>Total hot run time: 38205 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43766128fba016bea03e0940bfb596b1ef521ac3, data reload: false
------ Round 1 ----------------------------------
q1 17648 4943 4928 4928
q2 2011 149 141 141
q3 10608 1156 1169 1156
q4 10193 781 786 781
q5 7792 2970 2848 2848
q6 207 124 122 122
q7 906 505 492 492
q8 9273 1968 1977 1968
q9 6404 6360 6274 6274
q10 8283 3008 3002 3002
q11 446 209 208 208
q12 385 229 224 224
q13 17985 3350 3344 3344
q14 243 203 194 194
q15 538 513 506 506
q16 510 392 399 392
q17 942 676 525 525
q18 7361 6722 6649 6649
q19 1584 1507 1504 1504
q20 700 297 308 297
q21 2735 2435 2347 2347
q22 371 303 305 303
Total cold run time: 107125 ms
Total hot run time: 38205 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4918 4940 4889 4889
q2 336 230 253 230
q3 3272 3277 3268 3268
q4 2118 2022 1991 1991
q5 5763 5798 5776 5776
q6 209 119 119 119
q7 2300 1860 1925 1860
q8 3366 3451 3438 3438
q9 8804 8723 8735 8723
q10 3764 3848 3815 3815
q11 548 451 445 445
q12 795 614 635 614
q13 5560 3187 3163 3163
q14 281 267 255 255
q15 547 501 505 501
q16 556 493 506 493
q17 1886 1861 1857 1857
q18 8613 8394 8250 8250
q19 1591 1614 1623 1614
q20 2181 1963 1936 1936
q21 5532 5247 5191 5191
q22 521 437 469 437
Total cold run time: 63461 ms
Total hot run time: 58865 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880308107
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880401665
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.39 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.3 seconds inserted 10000000 Rows, about 366K ops/s
storage size: 17183932439 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880059661
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880070836
<details>
<summary>TPC-DS: <b>Total hot run time: 184742 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43766128fba016bea03e0940bfb596b1ef521ac3, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 948 347 354 347
query2 6761 2075 2011 2011
query3 6721 231 254 231
query4 27289 22626 22493 22493
query5 5849 607 670 607
query6 331 195 211 195
query7 4634 312 307 307
query8 257 238 223 223
query9 8846 2983 2989 2983
query10 559 324 317 317
query11 16369 15659 15676 15659
query12 169 97 95 95
query13 1675 342 332 332
query14 12647 7698 7835 7698
query15 292 218 228 218
query16 6412 354 331 331
query17 1697 538 526 526
query18 1969 300 303 300
query19 267 160 167 160
query20 107 108 100 100
query21 190 98 97 97
query22 4879 4795 4629 4629
query23 32526 31392 31554 31392
query24 12460 3103 3120 3103
query25 611 369 368 368
query26 1810 175 165 165
query27 2905 317 312 312
query28 6794 2125 2117 2117
query29 2106 417 421 417
query30 290 146 148 146
query31 1031 837 826 826
query32 142 85 86 85
query33 844 348 357 348
query34 961 496 484 484
query35 981 904 851 851
query36 1439 1239 1276 1239
query37 202 92 91 91
query38 3472 3384 3303 3303
query39 1352 1297 1310 1297
query40 303 91 92 91
query41 39 37 36 36
query42 106 110 104 104
query43 592 532 512 512
query44 1238 799 803 799
query45 214 214 214 214
query46 1172 736 744 736
query47 1717 1673 1640 1640
query48 386 294 287 287
query49 1194 339 330 330
query50 850 364 344 344
query51 5471 5353 5321 5321
query52 111 95 98 95
query53 246 170 167 167
query54 1461 685 689 685
query55 110 98 107 98
query56 285 262 240 240
query57 1075 979 971 971
query58 326 294 259 259
query59 3183 3033 2842 2842
query60 325 296 298 296
query61 148 146 151 146
query62 577 469 479 469
query63 211 166 168 166
query64 5791 1777 1730 1730
query65 3477 3367 3387 3367
query66 1358 387 384 384
query67 15361 15252 15028 15028
query68 12877 561 559 559
query69 645 334 319 319
query70 1789 1579 1590 1579
query71 619 280 298 280
query72 5261 3417 3451 3417
query73 2937 337 341 337
query74 7003 6492 6531 6492
query75 5382 2319 2333 2319
query76 6288 1220 1259 1220
query77 903 321 369 321
query78 9120 8748 8619 8619
query79 1126 548 551 548
query80 584 367 350 350
query81 490 216 218 216
query82 246 111 111 111
query83 238 152 146 146
query84 264 57 59 57
query85 957 277 277 277
query86 444 422 399 399
query87 3693 3448 3472 3448
query88 3001 2507 2499 2499
query89 377 303 296 296
query90 1945 230 249 230
query91 127 96 96 96
query92 87 78 84 78
query93 1193 489 429 429
query94 926 249 230 230
query95 568 520 519 519
query96 696 346 337 337
query97 4321 4234 4193 4193
query98 244 213 225 213
query99 1127 848 872 848
Total cold run time: 301599 ms
Total hot run time: 184742 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880059886
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880560012
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #29634:
URL: https://github.com/apache/doris/pull/29634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [rf](nereids)prune rf for external db according to jump count [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29634:
URL: https://github.com/apache/doris/pull/29634#issuecomment-1880069640
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.96 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 26.1 seconds inserted 10000000 Rows, about 383K ops/s
storage size: 17183534454 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org