You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/02/05 03:15:08 UTC
[PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
w41ter opened a new pull request, #30833:
URL: https://github.com/apache/doris/pull/30833
## Proposed changes
Issue Number: close #xxx
In cloud mode, both features are not supported.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926173894
<details>
<summary>TPC-H: <b>Total hot run time: 37526 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
------ Round 1 ----------------------------------
q1 18358 4728 4589 4589
q2 2623 145 149 145
q3 11140 948 943 943
q4 5021 752 744 744
q5 7805 2916 2940 2916
q6 187 123 125 123
q7 1151 773 744 744
q8 9703 2034 2030 2030
q9 7283 6368 6397 6368
q10 8107 2410 2443 2410
q11 426 217 206 206
q12 722 286 291 286
q13 17994 3281 3334 3281
q14 307 252 253 252
q15 539 492 488 488
q16 476 412 408 408
q17 945 532 499 499
q18 6745 6069 5984 5984
q19 1559 1380 1387 1380
q20 619 341 345 341
q21 7025 3201 3086 3086
q22 811 325 303 303
Total cold run time: 109546 ms
Total hot run time: 37526 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4516 4398 4472 4398
q2 332 238 234 234
q3 3014 2906 2778 2778
q4 1857 1622 1604 1604
q5 5205 5228 5271 5228
q6 192 114 115 114
q7 2140 1743 1819 1743
q8 3148 3264 3250 3250
q9 8364 8362 8345 8345
q10 5972 3540 3555 3540
q11 553 469 460 460
q12 763 576 573 573
q13 7034 3148 3132 3132
q14 285 244 263 244
q15 537 492 493 492
q16 536 484 489 484
q17 1842 1701 1742 1701
q18 7938 7733 7608 7608
q19 7152 1508 1494 1494
q20 2152 1925 1883 1883
q21 4803 4527 4573 4527
q22 577 470 472 470
Total cold run time: 68912 ms
Total hot run time: 54302 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926181786
<details>
<summary>TPC-DS: <b>Total hot run time: 174961 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
query1 923 348 342 342
query2 6524 2095 2122 2095
query3 6708 212 201 201
query4 31292 22129 22018 22018
query5 4308 429 357 357
query6 259 162 166 162
query7 4607 303 285 285
query8 238 175 178 175
query9 9120 2318 2318 2318
query10 416 218 204 204
query11 18853 15450 15318 15318
query12 130 73 72 72
query13 1612 423 436 423
query14 8729 6870 6783 6783
query15 234 171 179 171
query16 8101 252 253 252
query17 1859 533 497 497
query18 2103 287 262 262
query19 189 140 136 136
query20 81 73 76 73
query21 196 125 127 125
query22 4844 4693 4662 4662
query23 30923 30115 30125 30115
query24 11698 2776 2772 2772
query25 609 341 341 341
query26 1803 153 147 147
query27 3074 305 311 305
query28 7768 1870 1849 1849
query29 986 595 603 595
query30 289 133 140 133
query31 897 726 703 703
query32 88 52 53 52
query33 735 217 233 217
query34 1116 465 468 465
query35 860 776 763 763
query36 1025 926 951 926
query37 171 62 64 62
query38 3260 3096 3133 3096
query39 1306 1258 1259 1258
query40 271 92 98 92
query41 37 35 34 34
query42 106 91 93 91
query43 550 506 509 506
query44 1084 690 705 690
query45 191 182 171 171
query46 1066 656 646 646
query47 1583 1536 1535 1535
query48 411 355 354 354
query49 1216 289 288 288
query50 756 384 376 376
query51 5291 5200 5174 5174
query52 103 88 83 83
query53 336 272 271 271
query54 289 222 215 215
query55 78 75 74 74
query56 229 211 202 202
query57 1010 897 899 897
query58 215 182 173 173
query59 2442 2348 2468 2348
query60 246 216 210 210
query61 84 89 82 82
query62 665 349 341 341
query63 295 278 285 278
query64 6392 3718 3499 3499
query65 3262 3231 3206 3206
query66 1353 306 311 306
query67 14267 14165 14058 14058
query68 4290 517 540 517
query69 446 339 334 334
query70 1292 1261 1225 1225
query71 324 256 244 244
query72 5981 2838 2680 2680
query73 709 329 326 326
query74 6706 6239 6317 6239
query75 3014 2369 2333 2333
query76 2513 871 963 871
query77 316 226 233 226
query78 9100 8834 8650 8650
query79 2461 488 506 488
query80 1999 347 342 342
query81 520 196 191 191
query82 760 82 85 82
query83 246 130 124 124
query84 283 78 91 78
query85 2206 332 325 325
query86 484 294 307 294
query87 3377 3253 3222 3222
query88 3882 2318 2329 2318
query89 436 371 360 360
query90 2007 164 163 163
query91 155 120 121 120
query92 59 46 41 41
query93 2569 482 494 482
query94 1293 175 175 175
query95 7978 357 7776 357
query96 603 266 284 266
query97 4239 4120 4089 4089
query98 210 205 193 193
query99 1148 711 676 676
Total cold run time: 293368 ms
Total hot run time: 174961 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926162672
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926160209
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926174841
TeamCity be ut coverage result:
Function Coverage: 36.15% (8626/23861)
Line Coverage: 28.21% (70546/250066)
Region Coverage: 27.21% (36405/133779)
Branch Coverage: 24.00% (18642/77690)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c88fe0bf4be64762c91582d27b3df4ea436ec920_c88fe0bf4be64762c91582d27b3df4ea436ec920/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926185164
<details>
<summary>ClickBench: <b>Total hot run time: 30.09 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
query1 0.04 0.03 0.04
query2 0.06 0.02 0.02
query3 0.22 0.06 0.05
query4 1.68 0.10 0.09
query5 0.53 0.52 0.51
query6 1.18 0.65 0.66
query7 0.02 0.01 0.02
query8 0.04 0.03 0.02
query9 0.54 0.49 0.51
query10 0.54 0.55 0.55
query11 0.11 0.08 0.08
query12 0.11 0.08 0.09
query13 0.60 0.60 0.60
query14 0.77 0.79 0.80
query15 0.78 0.77 0.78
query16 0.39 0.38 0.38
query17 1.03 1.03 1.07
query18 0.22 0.28 0.24
query19 1.86 1.78 1.79
query20 0.02 0.01 0.01
query21 15.44 0.57 0.56
query22 2.59 1.71 1.78
query23 17.40 0.96 0.69
query24 2.71 1.66 0.26
query25 0.32 0.25 0.12
query26 0.58 0.14 0.15
query27 0.05 0.05 0.04
query28 11.60 0.83 0.85
query29 12.57 3.22 3.21
query30 0.64 0.53 0.54
query31 2.78 0.33 0.36
query32 3.38 0.48 0.49
query33 3.22 3.20 3.23
query34 15.76 4.26 4.24
query35 4.32 4.27 4.30
query36 1.10 1.05 1.07
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.16 0.13 0.12
query41 0.07 0.02 0.01
query42 0.03 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.62 s
Total hot run time: 30.09 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30833:
URL: https://github.com/apache/doris/pull/30833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926159865
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926166228
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926162695
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926187213
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.0 seconds inserted 10000000 Rows, about 714K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org