You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/07/04 15:01:13 UTC
[PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
luwei16 opened a new pull request, #37318:
URL: https://github.com/apache/doris/pull/37318
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210019724
<details>
<summary>TPC-DS: <b>Total hot run time: 174571 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
query1 931 374 364 364
query2 6458 2540 2433 2433
query3 6641 203 211 203
query4 28630 17533 17040 17040
query5 3674 493 498 493
query6 260 165 183 165
query7 4590 300 301 300
query8 325 303 294 294
query9 8634 2521 2483 2483
query10 596 307 287 287
query11 12679 9997 10028 9997
query12 129 92 85 85
query13 1655 379 385 379
query14 10328 7813 7958 7813
query15 242 188 192 188
query16 7090 331 320 320
query17 1767 566 560 560
query18 1684 285 282 282
query19 201 161 155 155
query20 92 82 82 82
query21 214 129 128 128
query22 4199 4211 3978 3978
query23 34037 33461 33594 33461
query24 11143 2826 2848 2826
query25 644 421 390 390
query26 1128 173 166 166
query27 2421 323 344 323
query28 7277 2226 2205 2205
query29 907 662 644 644
query30 263 165 161 161
query31 958 772 747 747
query32 96 53 61 53
query33 766 326 310 310
query34 961 507 491 491
query35 746 647 641 641
query36 1131 996 999 996
query37 160 79 86 79
query38 2975 2823 2780 2780
query39 888 877 836 836
query40 219 138 127 127
query41 55 56 53 53
query42 111 99 105 99
query43 602 561 540 540
query44 1229 727 726 726
query45 197 162 165 162
query46 1069 748 749 748
query47 1878 1779 1801 1779
query48 367 401 296 296
query49 859 405 415 405
query50 754 387 383 383
query51 6879 6758 6798 6758
query52 100 98 95 95
query53 367 286 296 286
query54 921 454 450 450
query55 78 72 73 72
query56 283 265 268 265
query57 1120 1073 1037 1037
query58 252 235 255 235
query59 3591 3610 3340 3340
query60 304 277 273 273
query61 100 98 95 95
query62 627 454 439 439
query63 319 292 292 292
query64 8910 2219 1803 1803
query65 3165 3090 3145 3090
query66 765 341 332 332
query67 15749 15147 15020 15020
query68 8495 558 568 558
query69 743 452 342 342
query70 1248 1171 1104 1104
query71 535 287 270 270
query72 8851 6287 5539 5539
query73 1793 324 325 324
query74 6036 5483 5533 5483
query75 5190 2679 2662 2662
query76 4879 991 1009 991
query77 773 305 293 293
query78 9514 8970 8837 8837
query79 9232 521 524 521
query80 1190 473 489 473
query81 580 226 225 225
query82 739 109 110 109
query83 327 190 163 163
query84 273 91 85 85
query85 1422 299 298 298
query86 397 327 314 314
query87 3278 3233 3132 3132
query88 4086 2457 2489 2457
query89 542 384 382 382
query90 2051 191 192 191
query91 132 103 103 103
query92 65 48 49 48
query93 6547 530 518 518
query94 1284 213 210 210
query95 408 311 329 311
query96 615 273 269 269
query97 3200 2995 3061 2995
query98 219 196 196 196
query99 1182 844 830 830
Total cold run time: 304216 ms
Total hot run time: 174571 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209188513
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #37318:
URL: https://github.com/apache/doris/pull/37318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209890004
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209966647
<details>
<summary>TPC-H: <b>Total hot run time: 39638 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
------ Round 1 ----------------------------------
q1 17606 4402 4276 4276
q2 2024 196 193 193
q3 10491 1184 1160 1160
q4 10203 794 772 772
q5 7480 2703 2593 2593
q6 222 135 133 133
q7 952 607 622 607
q8 9238 2062 2111 2062
q9 8841 6492 6492 6492
q10 8836 3632 3682 3632
q11 461 240 242 240
q12 435 233 224 224
q13 19070 2961 2975 2961
q14 265 243 235 235
q15 528 496 489 489
q16 502 375 373 373
q17 964 587 645 587
q18 7989 7477 7458 7458
q19 7112 1409 1342 1342
q20 660 326 324 324
q21 4892 3149 3225 3149
q22 381 344 336 336
Total cold run time: 119152 ms
Total hot run time: 39638 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4383 4215 4223 4215
q2 363 265 261 261
q3 2938 2926 2896 2896
q4 2057 1661 1754 1661
q5 5583 5500 5443 5443
q6 237 132 127 127
q7 2209 1904 1867 1867
q8 3297 3404 3438 3404
q9 8679 8721 8789 8721
q10 4101 3801 3690 3690
q11 584 516 499 499
q12 828 635 650 635
q13 15967 3130 3126 3126
q14 309 291 269 269
q15 541 474 475 474
q16 472 428 418 418
q17 1831 1516 1505 1505
q18 8009 8060 7980 7980
q19 1775 1481 1594 1481
q20 2166 1894 1865 1865
q21 5105 4981 4826 4826
q22 634 555 583 555
Total cold run time: 72068 ms
Total hot run time: 55918 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210975455
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37318:
URL: https://github.com/apache/doris/pull/37318#discussion_r1665833248
##########
be/test/olap/compaction_task_test.cpp:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock-actions.h>
Review Comment:
warning: 'gmock/gmock-actions.h' file not found [clang-diagnostic-error]
```cpp
#include <gmock/gmock-actions.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210043929
<details>
<summary>ClickBench: <b>Total hot run time: 31.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.03
query3 0.22 0.04 0.05
query4 1.67 0.07 0.07
query5 0.50 0.49 0.51
query6 1.14 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.55 0.50 0.51
query10 0.54 0.53 0.57
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.59 0.59 0.58
query14 0.77 0.78 0.78
query15 0.85 0.81 0.81
query16 0.36 0.37 0.35
query17 1.05 0.99 0.96
query18 0.25 0.22 0.27
query19 1.94 1.77 1.82
query20 0.01 0.01 0.02
query21 15.44 0.75 0.64
query22 4.16 7.22 2.38
query23 18.26 1.42 1.31
query24 2.12 0.24 0.22
query25 0.15 0.09 0.09
query26 0.30 0.21 0.21
query27 0.45 0.23 0.23
query28 13.28 1.02 0.99
query29 12.61 3.31 3.32
query30 0.25 0.06 0.06
query31 2.86 0.40 0.40
query32 3.25 0.47 0.48
query33 2.91 2.89 2.92
query34 17.08 4.31 4.32
query35 4.39 4.44 4.39
query36 0.66 0.48 0.48
query37 0.19 0.16 0.15
query38 0.16 0.14 0.14
query39 0.05 0.03 0.03
query40 0.14 0.13 0.12
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.84 s
Total hot run time: 31.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #37318:
URL: https://github.com/apache/doris/pull/37318#discussion_r1665841667
##########
be/src/olap/base_tablet.h:
##########
@@ -42,6 +43,8 @@ struct TabletWithVersion {
int64_t version;
};
+enum CompactionStage { NOT_SCHEDULED, PENDING, EXECUTING };
Review Comment:
enum class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210975415
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org