You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2024/09/27 09:51:30 UTC
[PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
mymeiyi opened a new pull request, #41413:
URL: https://github.com/apache/doris/pull/41413
now the cluster key does not support non vertical compaction.
https://github.com/apache/doris/pull/41208 try to implement it, but there are some problems, so we skip non vertical compaction for cluster key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378996866
<details>
<summary>ClickBench: <b>Total hot run time: 32.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false
query1 0.05 0.05 0.04
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.64 0.10 0.09
query5 0.51 0.52 0.53
query6 1.14 0.72 0.73
query7 0.02 0.01 0.01
query8 0.04 0.04 0.03
query9 0.59 0.52 0.51
query10 0.58 0.58 0.56
query11 0.14 0.11 0.11
query12 0.14 0.11 0.12
query13 0.61 0.60 0.60
query14 2.75 2.74 2.72
query15 0.90 0.83 0.82
query16 0.40 0.39 0.38
query17 1.04 1.04 1.00
query18 0.20 0.20 0.19
query19 1.87 1.82 2.03
query20 0.01 0.01 0.01
query21 15.35 0.55 0.55
query22 3.03 1.69 1.72
query23 17.21 0.97 0.78
query24 2.46 0.73 1.00
query25 0.22 0.08 0.17
query26 0.32 0.15 0.14
query27 0.05 0.05 0.04
query28 11.33 1.10 1.07
query29 12.52 3.24 3.23
query30 0.25 0.06 0.07
query31 2.88 0.39 0.38
query32 3.27 0.48 0.47
query33 3.04 3.02 3.07
query34 17.01 4.48 4.45
query35 4.52 4.44 4.50
query36 0.65 0.48 0.49
query37 0.08 0.06 0.06
query38 0.05 0.04 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.12
query41 0.07 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.04 0.02
Total cold run time: 107.48 s
Total hot run time: 32.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2405249022
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378944184
TeamCity be ut coverage result:
Function Coverage: 37.30% (9627/25811)
Line Coverage: 28.70% (79668/277598)
Region Coverage: 28.13% (41192/146446)
Branch Coverage: 24.74% (20974/84762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8c8b5df02569837e91c184417dbfc5276f6d9d4_a8c8b5df02569837e91c184417dbfc5276f6d9d4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378968112
<details>
<summary>TPC-H: <b>Total hot run time: 41004 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false
------ Round 1 ----------------------------------
q1 17586 7565 7328 7328
q2 2023 272 275 272
q3 12158 1101 1174 1101
q4 10562 722 782 722
q5 7755 2885 2809 2809
q6 237 156 154 154
q7 982 628 617 617
q8 9387 1941 1976 1941
q9 7564 6437 6415 6415
q10 6942 2276 2319 2276
q11 434 242 245 242
q12 429 219 221 219
q13 17786 2968 2965 2965
q14 246 209 212 209
q15 566 520 524 520
q16 658 588 594 588
q17 965 538 554 538
q18 7246 6675 6691 6675
q19 1386 1013 1124 1013
q20 466 196 204 196
q21 4005 3245 3214 3214
q22 1110 990 992 990
Total cold run time: 110493 ms
Total hot run time: 41004 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7312 7277 7312 7277
q2 334 226 234 226
q3 3070 2917 2881 2881
q4 2086 1825 1813 1813
q5 5810 5764 5747 5747
q6 245 142 145 142
q7 2206 1863 1772 1772
q8 3369 3562 3441 3441
q9 8931 8971 8854 8854
q10 3593 3588 3536 3536
q11 579 473 490 473
q12 825 586 668 586
q13 9996 3209 3194 3194
q14 296 281 283 281
q15 572 529 526 526
q16 693 659 648 648
q17 1856 1611 1600 1600
q18 8140 7764 7603 7603
q19 1696 1497 1548 1497
q20 2086 1870 1902 1870
q21 5584 5388 5510 5388
q22 1150 1039 1065 1039
Total cold run time: 70429 ms
Total hot run time: 60394 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2403937715
TeamCity be ut coverage result:
Function Coverage: 37.25% (9646/25894)
Line Coverage: 28.55% (79965/280118)
Region Coverage: 27.99% (41346/147742)
Branch Coverage: 24.60% (21051/85590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d39d0ea471cdcc46e139ff0472e8ca1d980e6e8_3d39d0ea471cdcc46e139ff0472e8ca1d980e6e8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378895095
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2381076403
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2403882292
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2403879408
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2404114276
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2405249130
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378893849
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41413:
URL: https://github.com/apache/doris/pull/41413#issuecomment-2378987435
<details>
<summary>TPC-DS: <b>Total hot run time: 191675 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false
query1 916 396 396 396
query2 6257 2074 2076 2074
query3 8677 195 205 195
query4 33577 23577 23486 23486
query5 3535 471 454 454
query6 266 165 168 165
query7 4201 302 312 302
query8 288 221 233 221
query9 9261 2670 2670 2670
query10 475 279 285 279
query11 17921 15289 15224 15224
query12 152 100 96 96
query13 1543 418 404 404
query14 9635 7544 7493 7493
query15 242 170 175 170
query16 7924 481 479 479
query17 1683 628 611 611
query18 1900 313 323 313
query19 388 165 159 159
query20 120 116 119 116
query21 223 102 112 102
query22 4905 4536 4503 4503
query23 35007 34130 33879 33879
query24 11011 2847 2815 2815
query25 639 410 404 404
query26 1135 158 156 156
query27 2273 298 298 298
query28 7502 2409 2390 2390
query29 796 434 423 423
query30 313 153 155 153
query31 1076 746 825 746
query32 99 57 58 57
query33 771 306 302 302
query34 925 503 518 503
query35 855 769 734 734
query36 1115 956 938 938
query37 150 86 85 85
query38 4004 3925 3891 3891
query39 1491 1432 1408 1408
query40 210 97 97 97
query41 52 47 49 47
query42 124 94 97 94
query43 547 465 473 465
query44 1217 799 800 799
query45 194 166 171 166
query46 1162 706 709 706
query47 1924 1777 1804 1777
query48 464 371 375 371
query49 969 413 407 407
query50 826 419 418 418
query51 7142 6937 6860 6860
query52 102 89 86 86
query53 247 181 184 181
query54 1214 471 468 468
query55 82 79 79 79
query56 299 247 248 247
query57 1187 1090 1080 1080
query58 234 217 227 217
query59 3172 2910 2963 2910
query60 289 268 276 268
query61 108 100 108 100
query62 865 661 670 661
query63 216 187 184 184
query64 4002 633 620 620
query65 3242 3160 3197 3160
query66 837 305 307 305
query67 15837 15509 15485 15485
query68 4596 562 554 554
query69 540 294 295 294
query70 1181 1118 1120 1118
query71 356 267 269 267
query72 7254 4114 4013 4013
query73 764 338 342 338
query74 10236 8950 8989 8950
query75 3469 2691 2642 2642
query76 2803 911 932 911
query77 617 299 294 294
query78 10619 9606 9513 9513
query79 2572 596 620 596
query80 2477 449 457 449
query81 585 239 239 239
query82 676 134 141 134
query83 325 133 139 133
query84 284 78 80 78
query85 1619 301 297 297
query86 456 304 297 297
query87 4425 4182 4306 4182
query88 4102 2390 2459 2390
query89 409 289 288 288
query90 2115 187 184 184
query91 179 141 163 141
query92 69 48 48 48
query93 2250 536 541 536
query94 1185 290 279 279
query95 352 254 254 254
query96 633 278 279 278
query97 3239 3126 3160 3126
query98 221 197 196 196
query99 1599 1304 1304 1304
Total cold run time: 302222 ms
Total hot run time: 191675 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #41413:
URL: https://github.com/apache/doris/pull/41413#discussion_r1794504799
##########
be/src/olap/compaction.cpp:
##########
@@ -186,7 +186,7 @@ Status Compaction::merge_input_rowsets() {
Status res;
{
SCOPED_TIMER(_merge_rowsets_latency_timer);
- if (_is_vertical) {
+ if (_is_vertical || !_tablet->tablet_schema()->cluster_key_idxes().empty()) {
Review Comment:
if non-vertical is configed and table has cluster key, just skip compaction by return error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cluster key) Cluster key skip non vertical compact [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #41413:
URL: https://github.com/apache/doris/pull/41413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org