You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2024/07/03 14:14:09 UTC
[PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
bobhan1 opened a new pull request, #37242:
URL: https://github.com/apache/doris/pull/37242
## Proposed changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206232953
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206723953
<details>
<summary>TPC-H: <b>Total hot run time: 40160 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 542cb8613689602489ad7e2095ef49c24a8a97c5, data reload: false
------ Round 1 ----------------------------------
q1 17706 4453 4353 4353
q2 2014 191 185 185
q3 10512 1252 1196 1196
q4 10222 746 782 746
q5 7497 2689 2592 2592
q6 217 137 144 137
q7 954 607 608 607
q8 9229 2071 2104 2071
q9 9236 6579 6514 6514
q10 9017 3754 3740 3740
q11 464 243 243 243
q12 525 244 246 244
q13 17762 3002 2998 2998
q14 261 231 227 227
q15 539 503 484 484
q16 524 378 378 378
q17 981 672 731 672
q18 8120 7457 7357 7357
q19 7892 1558 1483 1483
q20 673 334 345 334
q21 4976 3885 3270 3270
q22 393 329 338 329
Total cold run time: 119714 ms
Total hot run time: 40160 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4376 4267 4266 4266
q2 365 264 265 264
q3 3129 2896 2923 2896
q4 1997 1776 1700 1700
q5 5553 5482 5386 5386
q6 248 134 133 133
q7 2292 1836 1801 1801
q8 3227 3452 3393 3393
q9 8815 8817 8856 8817
q10 4226 3727 3781 3727
q11 604 500 495 495
q12 843 634 652 634
q13 16034 3158 3189 3158
q14 310 282 267 267
q15 549 473 501 473
q16 491 435 447 435
q17 1855 1518 1494 1494
q18 8202 7911 7852 7852
q19 1893 1524 1586 1524
q20 2997 1880 1859 1859
q21 5209 4963 5051 4963
q22 739 576 528 528
Total cold run time: 73954 ms
Total hot run time: 56065 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206264343
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #37242:
URL: https://github.com/apache/doris/pull/37242#discussion_r1665057806
##########
be/src/olap/base_tablet.cpp:
##########
@@ -1066,6 +1066,15 @@ void BaseTablet::add_sentinel_mark_to_delete_bitmap(DeleteBitmap* delete_bitmap,
}
}
+void BaseTablet::add_skip_mark_to_delete_bitmap(DeleteBitmap* delete_bitmap,
Review Comment:
you don't need to add a mark, just check the rowset's version, if the version is not single version, e.g [x-x], it means that the new rowset is generated by compaction, you can skip it directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206440713
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206752100
<details>
<summary>ClickBench: <b>Total hot run time: 30.75 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 542cb8613689602489ad7e2095ef49c24a8a97c5, data reload: false
query1 0.04 0.03 0.03
query2 0.09 0.04 0.04
query3 0.23 0.05 0.05
query4 1.66 0.07 0.08
query5 0.51 0.49 0.49
query6 1.14 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.56 0.49 0.48
query10 0.55 0.53 0.55
query11 0.14 0.11 0.12
query12 0.15 0.11 0.12
query13 0.59 0.58 0.59
query14 0.78 0.78 0.78
query15 0.85 0.82 0.81
query16 0.38 0.36 0.37
query17 1.01 1.00 1.01
query18 0.20 0.25 0.26
query19 1.87 1.71 1.79
query20 0.02 0.01 0.01
query21 15.39 0.77 0.65
query22 3.98 7.22 2.12
query23 18.32 1.34 1.29
query24 2.14 0.26 0.23
query25 0.16 0.09 0.08
query26 0.27 0.18 0.18
query27 0.08 0.09 0.08
query28 13.19 1.01 1.00
query29 12.67 3.27 3.19
query30 0.26 0.06 0.07
query31 2.88 0.39 0.39
query32 3.24 0.49 0.47
query33 2.95 2.83 2.94
query34 17.16 4.52 4.47
query35 4.50 4.53 4.50
query36 0.66 0.46 0.49
query37 0.19 0.15 0.16
query38 0.15 0.14 0.15
query39 0.05 0.03 0.04
query40 0.16 0.13 0.15
query41 0.10 0.04 0.05
query42 0.06 0.04 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.45 s
Total hot run time: 30.75 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37242:
URL: https://github.com/apache/doris/pull/37242#issuecomment-2206743098
<details>
<summary>TPC-DS: <b>Total hot run time: 173301 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 542cb8613689602489ad7e2095ef49c24a8a97c5, data reload: false
query1 917 384 374 374
query2 6449 2417 2289 2289
query3 6635 211 229 211
query4 18519 17425 17120 17120
query5 3618 495 516 495
query6 297 180 179 179
query7 4608 293 298 293
query8 302 273 278 273
query9 8422 2436 2409 2409
query10 557 303 282 282
query11 10546 9962 10090 9962
query12 115 88 86 86
query13 1649 376 384 376
query14 10514 7149 7636 7149
query15 230 190 181 181
query16 7775 318 335 318
query17 1700 580 544 544
query18 1964 299 279 279
query19 204 163 160 160
query20 90 82 88 82
query21 214 144 127 127
query22 4361 4185 4020 4020
query23 33724 33435 33415 33415
query24 11101 2858 2875 2858
query25 655 442 423 423
query26 722 181 169 169
query27 2299 332 338 332
query28 6225 2189 2167 2167
query29 918 692 661 661
query30 265 159 166 159
query31 995 765 753 753
query32 99 58 58 58
query33 805 311 316 311
query34 1081 517 499 499
query35 783 651 666 651
query36 1130 964 996 964
query37 139 81 84 81
query38 2997 2876 2807 2807
query39 911 856 840 840
query40 220 130 137 130
query41 56 50 52 50
query42 125 105 99 99
query43 562 564 560 560
query44 1204 763 733 733
query45 199 166 160 160
query46 1083 765 733 733
query47 1848 1740 1778 1740
query48 385 303 310 303
query49 839 428 423 423
query50 763 390 389 389
query51 6833 6834 6853 6834
query52 105 91 92 91
query53 360 294 292 292
query54 908 455 439 439
query55 74 74 73 73
query56 289 259 271 259
query57 1126 1057 1025 1025
query58 249 255 240 240
query59 3510 3287 3183 3183
query60 304 302 295 295
query61 116 92 94 92
query62 620 439 442 439
query63 328 287 287 287
query64 8535 2259 1791 1791
query65 3157 3239 3129 3129
query66 752 326 325 325
query67 15568 14922 14891 14891
query68 5942 567 555 555
query69 713 475 346 346
query70 1158 1169 1148 1148
query71 475 280 287 280
query72 9144 5532 5506 5506
query73 786 332 330 330
query74 5972 5483 5493 5483
query75 4589 2664 2707 2664
query76 4060 1042 928 928
query77 665 309 313 309
query78 9564 8895 8802 8802
query79 8328 514 510 510
query80 1161 468 490 468
query81 558 216 225 216
query82 749 110 100 100
query83 298 176 170 170
query84 269 87 86 86
query85 1341 291 272 272
query86 456 311 310 310
query87 3305 3126 3130 3126
query88 5072 2359 2360 2359
query89 550 389 382 382
query90 1913 195 191 191
query91 131 102 99 99
query92 67 48 49 48
query93 6327 507 504 504
query94 1200 212 215 212
query95 426 320 373 320
query96 596 268 263 263
query97 3155 2966 2951 2951
query98 220 203 195 195
query99 1145 806 836 806
Total cold run time: 284988 ms
Total hot run time: 173301 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 closed pull request #37242: [opt](merge-on-write) Skip considering rowsets produced by compaction from commited txns' rowsets in publish phase
URL: https://github.com/apache/doris/pull/37242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org