You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2024/01/25 08:50:16 UTC
[PR] [feature](load) enable memtable on sink node by default [doris]
kaijchen opened a new pull request, #30372:
URL: https://github.com/apache/doris/pull/30372
## Proposed changes
Set `enable_memtable_on_sink_node` to `true` by default.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #30372:
URL: https://github.com/apache/doris/pull/30372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1910141394
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1909757610
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.0 seconds inserted 10000000 Rows, about 714K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1909679714
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1909743416
<details>
<summary>TPC-DS: <b>Total hot run time: 185920 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false
query1 938 333 331 331
query2 6552 2003 1856 1856
query3 6691 205 199 199
query4 32416 22227 22062 22062
query5 4455 373 386 373
query6 239 154 164 154
query7 4597 253 251 251
query8 222 188 170 170
query9 8402 2616 2573 2573
query10 414 233 221 221
query11 16829 15618 15402 15402
query12 116 76 65 65
query13 1679 377 377 377
query14 10567 6985 6978 6978
query15 207 183 185 183
query16 5796 254 251 251
query17 951 462 479 462
query18 1791 257 244 244
query19 176 129 130 129
query20 64 68 68 68
query21 192 131 130 130
query22 4968 4858 4698 4698
query23 31567 30558 30803 30558
query24 12647 2819 2800 2800
query25 560 301 309 301
query26 1783 147 135 135
query27 3228 282 288 282
query28 7464 1824 1822 1822
query29 1983 623 627 623
query30 286 136 134 134
query31 925 729 778 729
query32 72 52 45 45
query33 695 207 214 207
query34 1161 461 459 459
query35 846 729 806 729
query36 1312 1220 1200 1200
query37 93 60 60 60
query38 3360 3259 3257 3257
query39 1315 1269 1238 1238
query40 341 86 83 83
query41 37 35 33 33
query42 86 77 80 77
query43 495 471 470 470
query44 1127 689 681 681
query45 190 174 170 170
query46 1063 652 640 640
query47 1610 1581 1562 1562
query48 385 307 311 307
query49 1191 290 281 281
query50 690 305 319 305
query51 5324 5211 5223 5211
query52 88 70 78 70
query53 313 253 249 249
query54 241 183 185 183
query55 79 75 71 71
query56 170 164 158 158
query57 1022 939 919 919
query58 183 152 152 152
query59 2903 2746 2668 2668
query60 204 173 175 173
query61 83 81 79 79
query62 588 356 356 356
query63 280 260 255 255
query64 6126 1786 1715 1715
query65 3331 3249 3245 3245
query66 1344 321 317 317
query67 15538 14986 14957 14957
query68 13157 534 516 516
query69 607 297 309 297
query70 1802 1524 1467 1467
query71 10473 10194 10192 10192
query72 4834 2818 2823 2818
query73 3219 315 316 315
query74 7028 6454 6397 6397
query75 5267 2311 2330 2311
query76 6684 1016 993 993
query77 687 230 219 219
query78 9047 8861 8742 8742
query79 1088 504 485 485
query80 546 319 308 308
query81 447 202 215 202
query82 202 77 83 77
query83 137 116 113 113
query84 273 69 67 67
query85 1077 322 315 315
query86 394 392 414 392
query87 3481 3322 3323 3322
query88 2933 2194 2174 2174
query89 441 349 347 347
query90 1989 183 179 179
query91 150 127 120 120
query92 49 43 41 41
query93 1552 451 413 413
query94 1235 154 154 154
query95 497 468 443 443
query96 632 325 313 313
query97 4221 4160 4199 4160
query98 193 194 192 192
query99 1014 693 726 693
Total cold run time: 307947 ms
Total hot run time: 185920 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1909752469
<details>
<summary>ClickBench: <b>Total hot run time: 30.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false
query1 0.03 0.02 0.02
query2 0.05 0.02 0.02
query3 0.22 0.05 0.05
query4 1.70 0.07 0.07
query5 0.54 0.53 0.52
query6 1.25 0.62 0.63
query7 0.02 0.02 0.01
query8 0.03 0.02 0.03
query9 0.54 0.49 0.50
query10 0.56 0.57 0.56
query11 0.11 0.08 0.08
query12 0.11 0.09 0.09
query13 0.61 0.60 0.60
query14 0.78 0.80 0.80
query15 0.78 0.78 0.78
query16 0.40 0.36 0.37
query17 1.01 1.02 1.01
query18 0.24 0.24 0.24
query19 1.89 1.83 1.82
query20 0.01 0.02 0.02
query21 15.42 0.60 0.56
query22 2.89 2.33 1.79
query23 17.33 0.79 0.72
query24 2.77 1.07 0.63
query25 0.39 0.11 0.05
query26 0.54 0.13 0.13
query27 0.07 0.06 0.05
query28 11.97 0.78 0.76
query29 12.49 3.09 3.16
query30 0.51 0.50 0.48
query31 2.78 0.35 0.34
query32 3.38 0.48 0.48
query33 3.17 3.25 3.22
query34 15.78 4.33 4.23
query35 4.29 4.31 4.28
query36 1.11 1.07 1.07
query37 0.06 0.04 0.05
query38 0.03 0.03 0.03
query39 0.02 0.02 0.01
query40 0.15 0.12 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 106.14 s
Total hot run time: 30.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1909721905
<details>
<summary>TPC-H: <b>Total hot run time: 39466 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b0da9a88e174f3e4f28bc94e826a9ab700e7a9f, data reload: false
------ Round 1 ----------------------------------
q1 17600 5332 5299 5299
q2 2040 139 134 134
q3 10827 1180 1183 1180
q4 10379 822 809 809
q5 8108 3236 3285 3236
q6 204 122 124 122
q7 887 516 507 507
q8 9550 1963 1986 1963
q9 7649 6556 6471 6471
q10 8223 3095 3037 3037
q11 414 230 208 208
q12 355 189 191 189
q13 17979 3388 3345 3345
q14 247 220 215 215
q15 546 506 501 501
q16 439 377 362 362
q17 942 597 532 532
q18 7750 7055 6904 6904
q19 1565 1395 1384 1384
q20 561 316 306 306
q21 2787 2474 2459 2459
q22 348 303 313 303
Total cold run time: 109400 ms
Total hot run time: 39466 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5386 5337 5220 5220
q2 323 217 207 207
q3 3365 3211 3210 3210
q4 2079 2061 2047 2047
q5 6126 5999 5996 5996
q6 197 112 118 112
q7 2302 1924 1898 1898
q8 3232 3397 3404 3397
q9 8900 8785 8829 8785
q10 3839 3840 3844 3840
q11 544 474 449 449
q12 822 645 607 607
q13 16896 3151 3156 3151
q14 269 254 260 254
q15 545 503 490 490
q16 513 454 461 454
q17 1844 1894 1835 1835
q18 9620 19022 9762 9762
q19 24207 1581 1494 1494
q20 4618 1954 1938 1938
q21 14462 5341 5389 5341
q22 956 525 514 514
Total cold run time: 111045 ms
Total hot run time: 61001 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](load) enable memtable on sink node by default [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30372:
URL: https://github.com/apache/doris/pull/30372#issuecomment-1910188995
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org