You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/03/12 08:17:55 UTC
[PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
luwei16 opened a new pull request, #32113:
URL: https://github.com/apache/doris/pull/32113
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991352139
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991012195
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991189399
TeamCity be ut coverage result:
Function Coverage: 34.97% (8555/24464)
Line Coverage: 26.81% (69524/259355)
Region Coverage: 26.05% (36073/138472)
Branch Coverage: 23.03% (18442/80094)
Coverage Report: http://coverage.selectdb-in.cc/coverage/41eaa0c67398ea3f69deb846528a26767b45ad54_41eaa0c67398ea3f69deb846528a26767b45ad54/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991011614
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991076137
<details>
<summary>ClickBench: <b>Total hot run time: 29.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
query1 0.04 0.04 0.04
query2 0.05 0.02 0.02
query3 0.22 0.06 0.06
query4 1.67 0.10 0.10
query5 0.51 0.51 0.52
query6 1.19 0.63 0.63
query7 0.02 0.02 0.01
query8 0.04 0.02 0.02
query9 0.56 0.48 0.49
query10 0.53 0.54 0.55
query11 0.12 0.09 0.09
query12 0.11 0.10 0.09
query13 0.57 0.57 0.57
query14 0.74 0.75 0.75
query15 0.78 0.78 0.77
query16 0.39 0.40 0.40
query17 0.98 1.00 0.99
query18 0.20 0.28 0.25
query19 1.77 1.75 1.71
query20 0.01 0.01 0.01
query21 15.42 0.56 0.54
query22 2.44 1.51 2.22
query23 17.13 0.87 0.72
query24 3.17 1.29 0.51
query25 0.20 0.08 0.03
query26 0.66 0.12 0.13
query27 0.03 0.03 0.04
query28 11.39 0.85 0.85
query29 12.59 3.25 3.22
query30 0.60 0.56 0.63
query31 2.80 0.34 0.33
query32 3.39 0.44 0.44
query33 2.82 2.79 2.80
query34 16.15 4.31 4.32
query35 4.40 4.28 4.27
query36 1.07 1.01 1.00
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.03 0.02 0.01
query40 0.17 0.13 0.13
query41 0.07 0.02 0.01
query42 0.03 0.02 0.02
query43 0.03 0.03 0.02
Total cold run time: 105.19 s
Total hot run time: 29.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32113:
URL: https://github.com/apache/doris/pull/32113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991082109
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 16.2 seconds inserted 10000000 Rows, about 617K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991046677
<details>
<summary>TPC-H: <b>Total hot run time: 36983 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
------ Round 1 ----------------------------------
q1 17643 4464 4257 4257
q2 3014 152 146 146
q3 11305 918 923 918
q4 5197 717 669 669
q5 7818 2900 2912 2900
q6 191 131 129 129
q7 1214 855 842 842
q8 9543 2109 2028 2028
q9 7368 6458 6446 6446
q10 8200 2467 2488 2467
q11 420 224 214 214
q12 788 293 286 286
q13 17988 3001 3011 3001
q14 279 247 259 247
q15 496 455 454 454
q16 480 394 413 394
q17 954 583 558 558
q18 6586 5873 5777 5777
q19 1578 1428 1473 1428
q20 551 306 298 298
q21 7384 3273 3232 3232
q22 787 292 303 292
Total cold run time: 109784 ms
Total hot run time: 36983 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4106 4067 4073 4067
q2 323 226 226 226
q3 2975 2837 2833 2833
q4 1817 1523 1531 1523
q5 5181 5231 5241 5231
q6 198 120 120 120
q7 2239 1863 1852 1852
q8 3152 3273 3254 3254
q9 8516 8542 8592 8542
q10 6177 3685 3631 3631
q11 537 440 432 432
q12 721 516 554 516
q13 8082 2854 2841 2841
q14 271 243 251 243
q15 485 448 440 440
q16 454 398 404 398
q17 1742 1465 1451 1451
q18 7674 7243 7137 7137
q19 6107 1501 1536 1501
q20 1908 1715 1701 1701
q21 4824 4777 4556 4556
q22 499 457 448 448
Total cold run time: 67988 ms
Total hot run time: 52943 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991066837
<details>
<summary>TPC-DS: <b>Total hot run time: 186625 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
query1 934 372 343 343
query2 7405 2080 2023 2023
query3 6714 215 210 210
query4 31083 20779 20552 20552
query5 4324 399 411 399
query6 266 172 167 167
query7 4634 305 295 295
query8 227 165 166 165
query9 9949 2306 2282 2282
query10 442 225 231 225
query11 16933 14102 14173 14102
query12 130 81 79 79
query13 1618 424 429 424
query14 11063 10849 10666 10666
query15 271 211 194 194
query16 8062 273 267 267
query17 1923 574 539 539
query18 2029 284 286 284
query19 243 158 156 156
query20 93 85 86 85
query21 203 124 127 124
query22 4613 4398 4383 4383
query23 31316 30844 30863 30844
query24 11440 2762 2838 2762
query25 632 378 384 378
query26 1682 155 159 155
query27 3080 345 348 345
query28 8088 1926 1908 1908
query29 998 660 625 625
query30 308 142 145 142
query31 978 719 722 719
query32 88 64 53 53
query33 760 252 249 249
query34 1071 464 464 464
query35 858 602 600 600
query36 993 872 892 872
query37 120 80 89 80
query38 3280 3145 3141 3141
query39 1427 1517 1404 1404
query40 289 110 111 110
query41 50 48 46 46
query42 103 93 104 93
query43 498 473 461 461
query44 1344 706 689 689
query45 280 266 254 254
query46 1088 679 685 679
query47 1643 1591 1613 1591
query48 445 394 357 357
query49 1198 332 326 326
query50 761 376 372 372
query51 6652 6546 6515 6515
query52 105 89 97 89
query53 341 284 318 284
query54 318 245 248 245
query55 84 85 84 84
query56 246 222 232 222
query57 1103 1038 1001 1001
query58 235 203 208 203
query59 2828 2641 2556 2556
query60 272 262 262 262
query61 113 112 114 112
query62 607 402 396 396
query63 296 282 271 271
query64 6638 3996 3986 3986
query65 3107 3027 3020 3020
query66 1392 349 317 317
query67 15151 14373 14222 14222
query68 8801 541 552 541
query69 620 375 382 375
query70 1275 1115 1059 1059
query71 514 269 276 269
query72 6696 2681 2483 2483
query73 745 320 324 320
query74 8509 6625 6681 6625
query75 9518 8054 7936 7936
query76 5284 962 900 900
query77 640 268 261 261
query78 10345 9579 9520 9520
query79 11809 504 511 504
query80 1810 366 370 366
query81 500 205 216 205
query82 659 196 205 196
query83 233 141 147 141
query84 284 84 72 72
query85 1278 319 307 307
query86 381 308 315 308
query87 3412 3284 3234 3234
query88 5031 2356 2363 2356
query89 510 356 358 356
query90 2061 178 175 175
query91 160 129 128 128
query92 52 46 48 46
query93 7348 510 498 498
query94 1276 174 171 171
query95 421 326 316 316
query96 594 276 265 265
query97 4421 4258 4253 4253
query98 237 216 211 211
query99 1146 762 760 760
Total cold run time: 324035 ms
Total hot run time: 186625 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991296848
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org