You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "XieJiann (via GitHub)" <gi...@apache.org> on 2024/01/17 08:53:13 UTC
[PR] [enhancement](Nereids): add builder for hyper graph [doris]
XieJiann opened a new pull request, #30061:
URL: https://github.com/apache/doris/pull/30061
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903566217
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener merged PR #30061:
URL: https://github.com/apache/doris/pull/30061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1905440147
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903313995
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903566240
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903352290
<details>
<summary>TPC-DS: <b>Total hot run time: 176950 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7096572dddec7695dac9d99dbf1722db952fd5aa, data reload: false
query1 927 336 327 327
query2 6623 1964 1867 1867
query3 6696 200 192 192
query4 29692 22287 22142 22142
query5 4460 369 433 369
query6 242 158 169 158
query7 4604 261 257 257
query8 221 172 172 172
query9 8294 2520 2503 2503
query10 437 230 251 230
query11 16158 15543 15432 15432
query12 121 67 62 62
query13 1681 370 371 370
query14 10576 6826 6946 6826
query15 215 176 175 175
query16 5784 262 251 251
query17 945 474 466 466
query18 1782 256 258 256
query19 173 134 131 131
query20 72 80 65 65
query21 184 138 128 128
query22 5138 4991 4837 4837
query23 31874 31139 31528 31139
query24 11908 2869 2844 2844
query25 593 331 317 317
query26 1723 154 152 152
query27 3246 308 311 308
query28 7559 1846 1827 1827
query29 2657 651 637 637
query30 756 147 167 147
query31 1567 761 778 761
query32 82 52 49 49
query33 700 218 215 215
query34 1285 469 482 469
query35 1347 806 796 796
query36 1264 1187 1236 1187
query37 107 61 76 61
query38 3452 3226 3317 3226
query39 1318 1253 1253 1253
query40 205 87 83 83
query41 36 35 34 34
query42 91 78 90 78
query43 478 487 456 456
query44 1080 687 681 681
query45 189 173 173 173
query46 1063 643 639 639
query47 1601 1548 1557 1548
query48 404 332 315 315
query49 1120 283 278 278
query50 697 310 311 310
query51 5276 5227 5203 5203
query52 89 75 74 74
query53 318 250 255 250
query54 243 190 185 185
query55 80 78 70 70
query56 177 164 178 164
query57 995 928 912 912
query58 189 158 159 158
query59 2820 2684 2682 2682
query60 209 192 182 182
query61 82 82 88 82
query62 611 360 364 360
query63 276 257 258 257
query64 4974 1767 1724 1724
query65 3326 3251 3219 3219
query66 1306 316 318 316
query67 15831 14954 15187 14954
query68 12722 535 506 506
query69 620 311 300 300
query70 1679 1522 1468 1468
query71 564 223 206 206
query72 4977 2836 2844 2836
query73 2880 314 314 314
query74 6915 6473 6506 6473
query75 5193 2331 2297 2297
query76 6748 1055 997 997
query77 655 228 235 228
query78 9597 8762 8741 8741
query79 1006 490 499 490
query80 544 328 312 312
query81 472 203 197 197
query82 201 81 77 77
query83 139 131 121 121
query84 274 68 68 68
query85 1069 337 313 313
query86 387 399 384 384
query87 3510 3377 3401 3377
query88 3168 2186 2190 2186
query89 448 354 350 350
query90 1975 185 184 184
query91 152 131 130 130
query92 56 44 41 41
query93 1333 458 456 456
query94 1185 158 156 156
query95 498 447 457 447
query96 613 318 320 318
query97 4254 4173 4125 4125
query98 199 195 180 180
query99 984 700 681 681
Total cold run time: 295246 ms
Total hot run time: 176950 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903357259
<details>
<summary>ClickBench: <b>Total hot run time: 30.78 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7096572dddec7695dac9d99dbf1722db952fd5aa, data reload: false
query1 0.03 0.03 0.02
query2 0.06 0.02 0.02
query3 0.22 0.05 0.05
query4 1.71 0.08 0.08
query5 0.54 0.51 0.53
query6 1.38 0.65 0.63
query7 0.02 0.02 0.01
query8 0.04 0.02 0.02
query9 0.55 0.50 0.51
query10 0.56 0.57 0.56
query11 0.11 0.08 0.09
query12 0.11 0.10 0.09
query13 0.60 0.61 0.61
query14 0.80 0.79 0.81
query15 0.79 0.76 0.78
query16 0.38 0.39 0.37
query17 0.99 0.98 1.00
query18 0.22 0.23 0.23
query19 1.74 1.70 1.74
query20 0.01 0.00 0.01
query21 15.40 0.54 0.57
query22 2.82 2.03 1.99
query23 17.13 0.76 0.67
query24 2.70 0.76 1.02
query25 0.44 0.24 0.22
query26 0.41 0.13 0.13
query27 0.08 0.05 0.04
query28 11.58 0.77 0.75
query29 12.54 3.15 3.24
query30 0.55 0.51 0.49
query31 2.78 0.34 0.34
query32 3.39 0.48 0.49
query33 3.19 3.23 3.23
query34 15.77 4.32 4.30
query35 4.29 4.36 4.35
query36 1.11 1.07 1.08
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.14 0.13
query41 0.07 0.01 0.02
query42 0.03 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.45 s
Total hot run time: 30.78 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1895359695
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903341564
<details>
<summary>TPC-H: <b>Total hot run time: 38704 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7096572dddec7695dac9d99dbf1722db952fd5aa, data reload: false
------ Round 1 ----------------------------------
q1 17627 5355 5305 5305
q2 2034 140 133 133
q3 10637 1120 1148 1120
q4 10235 773 802 773
q5 7776 3116 3109 3109
q6 196 118 117 117
q7 832 492 487 487
q8 9216 1920 1948 1920
q9 7264 6442 6342 6342
q10 8247 3023 3015 3015
q11 418 218 203 203
q12 351 187 181 181
q13 17992 3313 3362 3313
q14 247 219 218 218
q15 540 505 505 505
q16 431 394 371 371
q17 918 589 544 544
q18 7639 6963 6722 6722
q19 1540 1410 1379 1379
q20 572 298 301 298
q21 2724 2349 2391 2349
q22 352 300 315 300
Total cold run time: 107788 ms
Total hot run time: 38704 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5319 5422 5320 5320
q2 332 219 209 209
q3 3342 3215 3225 3215
q4 2057 2034 2099 2034
q5 5943 5925 5875 5875
q6 197 112 115 112
q7 2286 1869 1839 1839
q8 3212 3354 3373 3354
q9 8796 8745 8693 8693
q10 3909 3838 3859 3838
q11 558 449 448 448
q12 819 624 601 601
q13 16899 3129 3104 3104
q14 281 246 258 246
q15 551 505 497 497
q16 514 457 456 456
q17 1874 1845 1816 1816
q18 9464 16340 9511 9511
q19 20065 1585 1533 1533
q20 4577 1940 1929 1929
q21 14387 5700 5546 5546
q22 1004 552 543 543
Total cold run time: 106386 ms
Total hot run time: 60719 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1903360720
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 7096572dddec7695dac9d99dbf1722db952fd5aa with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.6 seconds inserted 10000000 Rows, about 684K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30061:
URL: https://github.com/apache/doris/pull/30061#discussion_r1461551248
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/JoinOrderJob.java:
##########
@@ -73,11 +73,12 @@ private Group optimizePlan(Group group) {
}
private Group optimizeJoin(Group group) {
- HyperGraph hyperGraph = HyperGraph.toDPhyperGraph(group);
- for (AbstractNode node : hyperGraph.getNodes()) {
+ HyperGraph.Builder builder = HyperGraph.builderForDPhyper(group);
+ for (AbstractNode node : builder.getNodes()) {
Review Comment:
builder.peekNodes maybe better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](Nereids): add builder for hyper graph [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30061:
URL: https://github.com/apache/doris/pull/30061#issuecomment-1895360484
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org