You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2023/12/06 12:37:25 UTC
[PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
yujun777 opened a new pull request, #28082:
URL: https://github.com/apache/doris/pull/28082
Problem:
When execute a sql query, FE will send a request to BE. The request contains tablets and their partitions' visible versions. BE need to obtains the tablet's rowsets in range [0, visible version]. But sometimes BE may had compacted visible version rowset with large version rowsets, so it can not return rowsets just in range [0, visible version]. Then BE will return error VERSION_ALREADY_MERGED -230.
Fix:
1. For cumulative and base compaction, BE should compact only those rowsets which version is equal or less than visible version. But if there are a lot of invisible versions, BE should also compact them to avoid use too memory;
2. When publish txn succ, FE will send request to BE to update partitions visible versions;
3. BE will send its partitions visible versions to FE every one minute. FE will diff them with latest visible versions, if BE's are fall behind, FE will send request to BE to update partitions visible versions;
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880531425
<details>
<summary>TPC-H: <b>Total hot run time: 38580 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bb85570fd3e52d6be663cf92fe3265f5d3e5c45, data reload: false
------ Round 1 ----------------------------------
q1 17689 4953 4973 4953
q2 2021 157 144 144
q3 10581 1159 1116 1116
q4 10198 800 835 800
q5 7808 3013 2964 2964
q6 200 123 122 122
q7 913 501 493 493
q8 9329 1994 2025 1994
q9 6840 6472 6362 6362
q10 8281 3049 3009 3009
q11 439 239 203 203
q12 385 225 222 222
q13 18296 3408 3450 3408
q14 239 209 202 202
q15 544 517 500 500
q16 462 402 390 390
q17 970 736 604 604
q18 7303 6718 6604 6604
q19 1565 1541 1514 1514
q20 734 309 302 302
q21 2743 2429 2378 2378
q22 359 296 311 296
Total cold run time: 107899 ms
Total hot run time: 38580 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4918 4906 4894 4894
q2 340 237 241 237
q3 3288 3318 3271 3271
q4 2120 2014 2018 2014
q5 5809 5821 5791 5791
q6 203 115 115 115
q7 2316 1891 1902 1891
q8 3360 3440 3478 3440
q9 8804 8710 8706 8706
q10 3805 3832 3837 3832
q11 546 440 447 440
q12 788 657 633 633
q13 3952 3215 3178 3178
q14 295 258 264 258
q15 540 502 505 502
q16 555 518 490 490
q17 1870 1864 1872 1864
q18 8618 8443 8250 8250
q19 1605 1637 1626 1626
q20 2215 1973 1950 1950
q21 5527 5198 5326 5198
q22 501 480 450 450
Total cold run time: 61975 ms
Total hot run time: 59030 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2003429722
TeamCity be ut coverage result:
Function Coverage: 35.24% (8700/24685)
Line Coverage: 27.08% (71202/262938)
Region Coverage: 26.32% (36919/140254)
Branch Coverage: 23.25% (18896/81278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ed9cc58d585dc876f7935a76ebec434ecefa263a_ed9cc58d585dc876f7935a76ebec434ecefa263a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1527981640
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -747,6 +747,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:167:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1846597268
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.25 seconds
stream load tsv: 597 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17217249846 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2068927223
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2066314709
TeamCity be ut coverage result:
Function Coverage: 35.41% (8921/25191)
Line Coverage: 27.12% (73301/270257)
Region Coverage: 26.26% (37863/144199)
Branch Coverage: 23.07% (19290/83628)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4c8a5d4b0607cc8e433dc667e9ecc03a8f4a6f5d_4c8a5d4b0607cc8e433dc667e9ecc03a8f4a6f5d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1911434461
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2042725981
TeamCity be ut coverage result:
Function Coverage: 35.58% (8888/24978)
Line Coverage: 27.34% (73023/267063)
Region Coverage: 26.50% (37729/142353)
Branch Coverage: 23.32% (19236/82490)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2284e6c9a6d3a86a7e0be94d6226409f187ca905_2284e6c9a6d3a86a7e0be94d6226409f187ca905/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2005971404
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1849562806
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1450608131
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -709,6 +709,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:163:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #28082:
URL: https://github.com/apache/doris/pull/28082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2081296143
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1846598364
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b03e29be54ca43ff721edb16b3f08040d3425f4f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4786 4524 4531 4524
q2 373 151 158 151
q3 1458 1251 1206 1206
q4 1102 940 912 912
q5 3145 3153 3139 3139
q6 258 127 127 127
q7 1000 478 479 478
q8 2219 2234 2188 2188
q9 6683 6693 6680 6680
q10 3198 3240 3250 3240
q11 318 207 205 205
q12 360 200 212 200
q13 4590 3820 3782 3782
q14 245 217 226 217
q15 575 519 531 519
q16 446 387 398 387
q17 998 574 587 574
q18 7401 7894 6967 6967
q19 1524 1329 1406 1329
q20 552 317 293 293
q21 3126 2705 2640 2640
q22 354 276 281 276
Total cold run time: 44711 ms
Total hot run time: 40034 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4481 4472 4459 4459
q2 267 162 172 162
q3 3552 3546 3535 3535
q4 2389 2386 2378 2378
q5 5763 5725 5742 5725
q6 244 120 122 120
q7 2400 1877 1851 1851
q8 3531 3540 3530 3530
q9 9072 9019 9019 9019
q10 3907 4017 4004 4004
q11 509 424 389 389
q12 775 602 597 597
q13 4311 3547 3553 3547
q14 282 262 248 248
q15 574 516 551 516
q16 509 462 487 462
q17 1874 1856 1870 1856
q18 8683 8206 8334 8206
q19 1745 1746 1785 1746
q20 2321 2039 2039 2039
q21 6770 6181 6213 6181
q22 500 418 443 418
Total cold run time: 64459 ms
Total hot run time: 60988 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880545867
<details>
<summary>TPC-DS: <b>Total hot run time: 183392 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bb85570fd3e52d6be663cf92fe3265f5d3e5c45, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 935 341 334 334
query2 6553 2081 1883 1883
query3 6711 222 221 221
query4 26181 22435 22408 22408
query5 5805 582 630 582
query6 334 207 205 205
query7 4639 312 300 300
query8 266 226 217 217
query9 8712 2957 2987 2957
query10 559 321 333 321
query11 16339 15733 15618 15618
query12 171 100 109 100
query13 1684 355 346 346
query14 13621 7692 7669 7669
query15 252 225 212 212
query16 6375 343 327 327
query17 1691 523 492 492
query18 1936 308 298 298
query19 232 157 161 157
query20 108 107 95 95
query21 185 90 94 90
query22 4983 4938 4784 4784
query23 32597 31191 31561 31191
query24 10651 2938 2942 2938
query25 606 363 355 355
query26 1416 170 171 170
query27 2838 322 319 319
query28 6802 2148 2139 2139
query29 1103 433 403 403
query30 289 143 147 143
query31 1026 816 827 816
query32 129 87 84 84
query33 843 341 347 341
query34 854 470 469 469
query35 1015 883 867 867
query36 1455 1241 1342 1241
query37 126 80 85 80
query38 3492 3345 3320 3320
query39 1359 1300 1289 1289
query40 183 91 85 85
query41 42 36 35 35
query42 114 100 110 100
query43 528 498 538 498
query44 1167 784 804 784
query45 225 199 203 199
query46 1054 706 698 698
query47 1731 1647 1606 1606
query48 368 284 280 280
query49 1077 310 305 305
query50 738 325 333 325
query51 5457 5416 5274 5274
query52 107 104 95 95
query53 230 171 166 166
query54 1429 648 669 648
query55 103 93 95 93
query56 274 269 254 254
query57 1035 996 975 975
query58 301 285 296 285
query59 2923 2874 2899 2874
query60 313 290 300 290
query61 129 133 145 133
query62 592 474 487 474
query63 184 174 170 170
query64 4964 1770 1689 1689
query65 3446 3345 3354 3345
query66 1078 380 413 380
query67 15741 15688 14800 14800
query68 16637 564 519 519
query69 592 323 325 323
query70 2349 1606 1554 1554
query71 581 291 299 291
query72 5400 3435 3447 3435
query73 4518 354 338 338
query74 6992 6530 6639 6530
query75 5591 2289 2259 2259
query76 6566 1137 1233 1137
query77 1177 324 304 304
query78 9139 8516 8517 8516
query79 2584 526 524 524
query80 530 338 345 338
query81 475 214 215 214
query82 225 104 107 104
query83 209 143 142 142
query84 253 63 58 58
query85 725 276 274 274
query86 424 433 441 433
query87 3645 3487 3495 3487
query88 3078 2498 2543 2498
query89 368 300 281 281
query90 2085 256 250 250
query91 139 94 95 94
query92 81 68 81 68
query93 1308 431 426 426
query94 944 243 253 243
query95 568 500 502 500
query96 643 330 336 330
query97 4275 4209 4220 4209
query98 234 233 217 217
query99 1154 877 855 855
Total cold run time: 303895 ms
Total hot run time: 183392 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1444162948
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -721,6 +721,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:165:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1890257213
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1925565312
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1911423779
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889579984
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880418851
TeamCity be ut coverage result:
Function Coverage: 36.62% (8629/23565)
Line Coverage: 28.64% (70135/244885)
Region Coverage: 27.59% (36262/131455)
Branch Coverage: 24.27% (18535/76366)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6bb85570fd3e52d6be663cf92fe3265f5d3e5c45_6bb85570fd3e52d6be663cf92fe3265f5d3e5c45/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2003201850
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1417229758
##########
be/src/olap/olap_common.h:
##########
@@ -20,6 +20,7 @@
#include <gen_cpp/Types_types.h>
Review Comment:
warning: 'gen_cpp/Types_types.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/Types_types.h>
^
```
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -701,6 +701,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:159:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1581998838
##########
fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java:
##########
@@ -60,14 +61,18 @@ public PublishVersionDaemon() {
@Override
protected void runAfterCatalogReady() {
+ Map<Long, Long> partitionVisibleVersions = Maps.newHashMap();
+ Map<Long, Set<Long>> backendPartitions = Maps.newHashMap();
+
try {
- publishVersion();
+ publishVersion(partitionVisibleVersions, backendPartitions);
+ sendBackendVisibleVersion(partitionVisibleVersions, backendPartitions);
Review Comment:
publish request hard to get visible version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1581998717
##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/ReplicasProcNode.java:
##########
@@ -44,7 +44,7 @@ public class ReplicasProcNode implements ProcNodeInterface {
.add("BackendId").add("Version").add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
.add("SchemaHash").add("LocalDataSize").add("RemoteDataSize").add("RowCount").add("State").add("IsBad")
.add("IsUserDrop")
- .add("VersionCount").add("PathHash").add("Path")
+ .add("VisibleVersionCount").add("TotalVersionCount").add("PathHash").add("Path")
Review Comment:
update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2081317210
<details>
<summary>TPC-DS: <b>Total hot run time: 187078 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da9c446bb52560c5f384361cdcb43c5a150b2037, data reload: false
query1 915 357 355 355
query2 6283 2345 2347 2345
query3 6640 199 205 199
query4 23090 21868 21985 21868
query5 3814 412 426 412
query6 261 178 192 178
query7 4642 319 287 287
query8 234 176 181 176
query9 8403 2389 2375 2375
query10 414 258 270 258
query11 15272 14733 14786 14733
query12 124 93 89 89
query13 1640 376 377 376
query14 10506 7656 7570 7570
query15 250 172 164 164
query16 8146 256 256 256
query17 1707 548 533 533
query18 2087 269 276 269
query19 304 143 148 143
query20 86 81 79 79
query21 197 130 119 119
query22 5027 4761 4702 4702
query23 33774 32965 33239 32965
query24 10528 2989 2920 2920
query25 604 370 372 370
query26 1121 162 149 149
query27 2485 351 341 341
query28 6977 2011 2008 2008
query29 845 600 595 595
query30 245 148 149 148
query31 953 726 713 713
query32 98 50 50 50
query33 731 240 242 240
query34 1030 472 462 462
query35 803 654 669 654
query36 1023 890 920 890
query37 127 64 66 64
query38 3097 3016 2977 2977
query39 1558 1568 1542 1542
query40 203 126 123 123
query41 41 40 37 37
query42 104 99 95 95
query43 574 545 546 545
query44 1293 730 726 726
query45 253 292 265 265
query46 1101 726 729 726
query47 1911 1874 1847 1847
query48 373 297 292 292
query49 867 391 389 389
query50 755 377 383 377
query51 6651 6706 6702 6702
query52 103 95 90 90
query53 343 276 280 276
query54 299 240 233 233
query55 77 68 72 68
query56 242 222 236 222
query57 1190 1143 1121 1121
query58 218 195 197 195
query59 3372 3203 3133 3133
query60 250 234 235 234
query61 92 89 88 88
query62 652 444 445 444
query63 302 280 273 273
query64 8469 7208 7149 7149
query65 3130 3027 3062 3027
query66 1363 336 330 330
query67 15308 15274 15034 15034
query68 5210 529 600 529
query69 489 300 303 300
query70 1087 1092 1051 1051
query71 426 269 273 269
query72 7235 2626 2432 2432
query73 709 320 316 316
query74 6461 6182 6167 6167
query75 3404 2663 2742 2663
query76 3295 923 1000 923
query77 371 265 267 265
query78 10881 10221 10130 10130
query79 3682 529 505 505
query80 1887 440 450 440
query81 542 220 223 220
query82 748 92 94 92
query83 323 177 172 172
query84 266 95 82 82
query85 1919 267 271 267
query86 470 295 285 285
query87 3278 3124 3087 3087
query88 4692 2339 2338 2338
query89 480 361 364 361
query90 2011 185 182 182
query91 123 99 98 98
query92 64 49 48 48
query93 4866 499 499 499
query94 1199 180 180 180
query95 393 311 309 309
query96 591 267 260 260
query97 3184 2945 2943 2943
query98 240 219 216 216
query99 1246 862 834 834
Total cold run time: 287349 ms
Total hot run time: 187078 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2076178623
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1850131722
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.59 seconds
stream load tsv: 592 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17219403487 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880360890
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889571865
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1450597129
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -709,6 +709,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
+ size_t count = 0;
+ for (const auto& rs_meta : _rs_metas) {
+ if (!(range.first > rs_meta->version().second || range.second < rs_meta->version().first)) {
+ count++;
+ }
+ }
+ return count;
+}
+
Status TabletMeta::add_rs_meta(const RowsetMetaSharedPtr& rs_meta) {
// check RowsetMeta is valid
for (auto& rs : _rs_metas) {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:165:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2066152821
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2069061052
<details>
<summary>TPC-H: <b>Total hot run time: 39165 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c421323f9bc91a5e7ff478b49b125b2507b8fc9d, data reload: false
------ Round 1 ----------------------------------
q1 17647 4778 4212 4212
q2 2021 194 194 194
q3 10898 1184 1235 1184
q4 10390 755 844 755
q5 7601 2836 2735 2735
q6 227 138 141 138
q7 1041 628 618 618
q8 9398 2135 2109 2109
q9 7720 6768 6716 6716
q10 8743 3641 3631 3631
q11 458 257 236 236
q12 539 225 220 220
q13 19910 3037 3048 3037
q14 293 256 237 237
q15 540 498 507 498
q16 559 383 371 371
q17 981 648 695 648
q18 7379 6837 6724 6724
q19 3162 1535 1546 1535
q20 658 334 306 306
q21 3523 2751 2841 2751
q22 367 310 315 310
Total cold run time: 114055 ms
Total hot run time: 39165 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4378 4184 4170 4170
q2 363 271 272 271
q3 2951 2761 2764 2761
q4 1865 1650 1646 1646
q5 5318 5326 5349 5326
q6 213 123 126 123
q7 2246 1841 1887 1841
q8 3226 3362 3370 3362
q9 8662 8619 8683 8619
q10 3928 3784 3651 3651
q11 574 478 485 478
q12 778 577 584 577
q13 16266 2931 2952 2931
q14 299 275 270 270
q15 505 491 477 477
q16 484 421 436 421
q17 1778 1477 1483 1477
q18 7545 7465 7442 7442
q19 1642 1542 1577 1542
q20 1940 1773 1735 1735
q21 4818 4776 4883 4776
q22 506 465 441 441
Total cold run time: 70285 ms
Total hot run time: 54337 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1581150129
##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/ReplicasProcNode.java:
##########
@@ -44,7 +44,7 @@ public class ReplicasProcNode implements ProcNodeInterface {
.add("BackendId").add("Version").add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
.add("SchemaHash").add("LocalDataSize").add("RemoteDataSize").add("RowCount").add("State").add("IsBad")
.add("IsUserDrop")
- .add("VersionCount").add("PathHash").add("Path")
+ .add("VisibleVersionCount").add("TotalVersionCount").add("PathHash").add("Path")
Review Comment:
We should not change name known by users, there may be some tools on it.
##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletsProcDir.java:
##########
@@ -52,9 +52,10 @@ public class TabletsProcDir implements ProcDirInterface {
.add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
.add("LocalDataSize").add("RemoteDataSize").add("RowCount").add("State")
.add("LstConsistencyCheckTime").add("CheckVersion")
- .add("VersionCount").add("QueryHits").add("PathHash").add("Path")
+ .add("VisibleVersionCount").add("TotalVersionCount").add("QueryHits").add("PathHash").add("Path")
Review Comment:
We should not change name already exposed to user.
##########
fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java:
##########
@@ -60,14 +61,18 @@ public PublishVersionDaemon() {
@Override
protected void runAfterCatalogReady() {
+ Map<Long, Long> partitionVisibleVersions = Maps.newHashMap();
+ Map<Long, Set<Long>> backendPartitions = Maps.newHashMap();
+
try {
- publishVersion();
+ publishVersion(partitionVisibleVersions, backendPartitions);
+ sendBackendVisibleVersion(partitionVisibleVersions, backendPartitions);
Review Comment:
If report handles it, we should skip send here. We can pack visible version in publish request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1467199113
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -720,6 +720,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
Review Comment:
warning: method 'version_count_cross_with_range' can be made static [readability-convert-member-functions-to-static]
```suggestion
size_t TabletMeta::version_count_cross_with_range(const Version& range) {
```
be/src/olap/tablet_meta.h:164:
```diff
- size_t version_count_cross_with_range(const Version& range) const;
+ static size_t version_count_cross_with_range(const Version& range) ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2042523848
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1850107279
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 9f6a48acc735a671afdf4443cb3cbbae624528d3, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4643 4467 4471 4467
q2 363 149 162 149
q3 1444 1251 1228 1228
q4 1108 983 958 958
q5 3125 3123 3154 3123
q6 253 128 127 127
q7 1027 491 479 479
q8 2210 2259 2191 2191
q9 6635 6711 6653 6653
q10 3186 3244 3264 3244
q11 322 206 205 205
q12 351 206 208 206
q13 4564 3990 3802 3802
q14 241 213 210 210
q15 564 516 524 516
q16 444 388 392 388
q17 1015 605 570 570
q18 7618 7539 7633 7539
q19 1508 1325 1477 1325
q20 598 317 299 299
q21 3063 2712 2698 2698
q22 355 278 282 278
Total cold run time: 44637 ms
Total hot run time: 40655 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4421 4409 4374 4374
q2 273 167 179 167
q3 3559 3542 3530 3530
q4 2383 2367 2379 2367
q5 5723 5736 5731 5731
q6 237 121 121 121
q7 2399 1883 1893 1883
q8 3525 3517 3525 3517
q9 9040 9007 9002 9002
q10 3927 3976 4007 3976
q11 508 380 391 380
q12 777 593 605 593
q13 4292 3544 3540 3540
q14 280 271 270 270
q15 575 525 505 505
q16 501 457 495 457
q17 1882 1857 1865 1857
q18 8696 8369 8414 8369
q19 1726 1771 1763 1763
q20 2248 1951 1928 1928
q21 6533 6170 6172 6170
q22 503 414 423 414
Total cold run time: 64008 ms
Total hot run time: 60914 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1846524173
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1848575997
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 29ca41ebc93af1bed3d3f60248bd339cc63276fc, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4625 4473 4488 4473
q2 364 177 152 152
q3 1453 1341 1198 1198
q4 1108 917 886 886
q5 3126 3136 3102 3102
q6 251 128 128 128
q7 990 476 474 474
q8 2168 2220 2161 2161
q9 6652 6619 6661 6619
q10 3192 3259 3252 3252
q11 326 207 205 205
q12 349 214 209 209
q13 4545 3785 3800 3785
q14 244 215 215 215
q15 561 525 520 520
q16 440 386 388 386
q17 983 591 503 503
q18 7490 7468 7006 7006
q19 1516 1290 1407 1290
q20 521 284 322 284
q21 3064 2613 2635 2613
q22 344 279 282 279
Total cold run time: 44312 ms
Total hot run time: 39740 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4408 4381 4403 4381
q2 265 164 175 164
q3 3524 3511 3514 3511
q4 2372 2361 2357 2357
q5 5736 5815 5685 5685
q6 242 123 122 122
q7 2348 1874 1899 1874
q8 3474 3485 3481 3481
q9 8985 8939 9034 8939
q10 3905 3968 3989 3968
q11 512 386 377 377
q12 769 621 596 596
q13 4281 3574 3547 3547
q14 295 250 266 250
q15 578 521 527 521
q16 500 446 474 446
q17 1850 1866 1884 1866
q18 8679 8299 8326 8299
q19 1710 1733 1750 1733
q20 2259 1941 1934 1934
q21 6502 6169 6135 6135
q22 495 417 435 417
Total cold run time: 63689 ms
Total hot run time: 60603 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1849772887
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.76 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17224122858 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2081322709
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2081316149
TeamCity be ut coverage result:
Function Coverage: 35.50% (8923/25132)
Line Coverage: 27.13% (73445/270688)
Region Coverage: 26.33% (37946/144117)
Branch Coverage: 23.11% (19326/83618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/da9c446bb52560c5f384361cdcb43c5a150b2037_da9c446bb52560c5f384361cdcb43c5a150b2037/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2069094802
<details>
<summary>ClickBench: <b>Total hot run time: 30.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c421323f9bc91a5e7ff478b49b125b2507b8fc9d, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.04
query4 1.69 0.07 0.08
query5 0.51 0.49 0.49
query6 1.47 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.56 0.49 0.50
query10 0.54 0.55 0.54
query11 0.16 0.12 0.11
query12 0.15 0.12 0.13
query13 0.62 0.58 0.58
query14 0.76 0.75 0.78
query15 0.83 0.80 0.81
query16 0.37 0.35 0.36
query17 1.02 0.96 1.02
query18 0.24 0.25 0.22
query19 1.77 1.68 1.69
query20 0.01 0.01 0.01
query21 15.45 0.65 0.64
query22 4.02 6.82 1.92
query23 18.23 1.34 1.27
query24 1.82 0.27 0.22
query25 0.16 0.09 0.08
query26 0.27 0.16 0.16
query27 0.07 0.08 0.07
query28 13.31 0.99 0.98
query29 12.54 3.27 3.24
query30 0.26 0.07 0.06
query31 2.87 0.38 0.36
query32 3.29 0.47 0.46
query33 2.84 2.84 2.76
query34 17.15 4.34 4.42
query35 4.47 4.52 4.51
query36 0.65 0.47 0.45
query37 0.17 0.16 0.15
query38 0.15 0.14 0.14
query39 0.05 0.03 0.03
query40 0.17 0.14 0.13
query41 0.09 0.04 0.04
query42 0.06 0.04 0.04
query43 0.04 0.04 0.03
Total cold run time: 109.25 s
Total hot run time: 30.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880458082
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.74 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.4 seconds inserted 10000000 Rows, about 364K ops/s
storage size: 17183928215 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1887597257
Why does the stale rowset MVCC mechanism not work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1848435579
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1849990538
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2081322702
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2066297064
<details>
<summary>TPC-H: <b>Total hot run time: 38811 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c8a5d4b0607cc8e433dc667e9ecc03a8f4a6f5d, data reload: false
------ Round 1 ----------------------------------
q1 17622 4596 4485 4485
q2 1989 180 184 180
q3 10461 1124 1203 1124
q4 10185 756 775 756
q5 7511 2709 2622 2622
q6 216 129 134 129
q7 1022 617 581 581
q8 9227 2081 2065 2065
q9 7627 6617 6544 6544
q10 8665 3553 3579 3553
q11 502 231 225 225
q12 490 222 217 217
q13 17782 2997 2935 2935
q14 272 234 224 224
q15 514 478 481 478
q16 544 411 378 378
q17 973 696 746 696
q18 7529 6817 6776 6776
q19 7397 1547 1492 1492
q20 657 317 304 304
q21 3481 2737 2909 2737
q22 359 313 310 310
Total cold run time: 115025 ms
Total hot run time: 38811 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4374 4247 4216 4216
q2 375 268 270 268
q3 2955 2695 2704 2695
q4 1872 1594 1577 1577
q5 5288 5312 5301 5301
q6 208 126 126 126
q7 2193 1875 1855 1855
q8 3220 3362 3376 3362
q9 8580 8605 8603 8603
q10 3890 3665 3703 3665
q11 574 479 471 471
q12 717 595 587 587
q13 17450 2975 2932 2932
q14 280 275 267 267
q15 512 473 472 472
q16 489 405 417 405
q17 1780 1485 1457 1457
q18 8002 7935 8005 7935
q19 1677 1612 1594 1594
q20 2055 1917 1801 1801
q21 8283 5060 5009 5009
q22 557 494 456 456
Total cold run time: 75331 ms
Total hot run time: 55054 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2069084478
<details>
<summary>TPC-DS: <b>Total hot run time: 185899 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c421323f9bc91a5e7ff478b49b125b2507b8fc9d, data reload: false
query1 916 379 369 369
query2 6486 2481 2626 2481
query3 6661 200 201 200
query4 23906 21321 21330 21321
query5 4171 411 434 411
query6 268 167 173 167
query7 4578 288 288 288
query8 239 180 189 180
query9 8587 2344 2346 2344
query10 595 266 261 261
query11 14750 14141 14214 14141
query12 138 91 88 88
query13 1634 360 343 343
query14 9830 7984 7429 7429
query15 261 191 182 182
query16 8078 251 251 251
query17 1942 574 552 552
query18 1932 267 264 264
query19 320 141 148 141
query20 91 83 84 83
query21 200 127 127 127
query22 5054 4820 4883 4820
query23 33817 33257 33030 33030
query24 12015 2949 3037 2949
query25 647 364 374 364
query26 1706 152 149 149
query27 2849 322 304 304
query28 7480 2015 2010 2010
query29 980 588 600 588
query30 309 172 178 172
query31 974 751 733 733
query32 94 51 53 51
query33 756 260 240 240
query34 1008 461 470 461
query35 840 697 684 684
query36 1042 906 915 906
query37 185 69 71 69
query38 3293 3206 3149 3149
query39 1593 1521 1505 1505
query40 280 125 121 121
query41 46 44 43 43
query42 99 96 98 96
query43 592 548 560 548
query44 1203 736 724 724
query45 286 259 262 259
query46 1088 706 758 706
query47 1945 1897 1902 1897
query48 359 284 295 284
query49 1173 389 399 389
query50 761 381 380 380
query51 6845 6765 6704 6704
query52 100 91 93 91
query53 355 277 279 277
query54 326 241 233 233
query55 75 72 74 72
query56 252 244 228 228
query57 1222 1124 1149 1124
query58 232 200 199 199
query59 3434 3414 3254 3254
query60 253 245 242 242
query61 113 107 107 107
query62 664 436 438 436
query63 301 274 274 274
query64 6508 4042 4060 4042
query65 3158 3066 3055 3055
query66 1360 344 347 344
query67 15462 15261 15137 15137
query68 5462 522 533 522
query69 518 310 309 309
query70 1233 1118 1145 1118
query71 1415 1264 1263 1263
query72 6564 2725 2550 2550
query73 742 318 320 318
query74 6951 6330 6480 6330
query75 3596 2683 2648 2648
query76 4015 979 1070 979
query77 623 270 269 269
query78 10952 10262 10146 10146
query79 8963 517 508 508
query80 1766 445 446 445
query81 523 241 245 241
query82 1567 94 96 94
query83 204 168 168 168
query84 267 89 86 86
query85 1774 300 273 273
query86 469 307 293 293
query87 3439 3305 3235 3235
query88 5315 2318 2332 2318
query89 553 372 358 358
query90 1975 174 174 174
query91 128 107 111 107
query92 64 45 44 44
query93 7398 507 505 505
query94 1208 176 175 175
query95 1093 1090 1091 1090
query96 613 265 261 261
query97 3182 2916 2946 2916
query98 238 227 219 219
query99 1273 864 870 864
Total cold run time: 302800 ms
Total hot run time: 185899 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2071257144
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2076284987
TeamCity be ut coverage result:
Function Coverage: 35.19% (8916/25335)
Line Coverage: 26.97% (73333/271877)
Region Coverage: 26.15% (37888/144876)
Branch Coverage: 22.97% (19293/83994)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3a668bc000562ed76b85844bff6d1bc392455ccf_3a668bc000562ed76b85844bff6d1bc392455ccf/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2076629706
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889501393
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1890273047
TeamCity be ut coverage result:
Function Coverage: 36.67% (8643/23568)
Line Coverage: 28.70% (70405/245318)
Region Coverage: 27.62% (36396/131765)
Branch Coverage: 24.34% (18619/76490)
Coverage Report: http://coverage.selectdb-in.cc/coverage/885e9922ff9ed87c6f424ff51c15f7fa2372ce29_885e9922ff9ed87c6f424ff51c15f7fa2372ce29/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889553644
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1880363168
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1842787086
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1842800653
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1849768175
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 13ab608280b019e96ff3933eec4b7d7c7ae9e6f9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4694 4437 4472 4437
q2 362 152 158 152
q3 1448 1244 1223 1223
q4 1100 921 916 916
q5 3150 3136 3148 3136
q6 245 126 127 126
q7 976 502 479 479
q8 2207 2220 2191 2191
q9 6692 6661 6762 6661
q10 3404 3220 3248 3220
q11 337 208 202 202
q12 347 204 213 204
q13 4610 3849 4185 3849
q14 242 212 220 212
q15 572 527 522 522
q16 446 391 390 390
q17 1007 607 580 580
q18 7579 7699 7187 7187
q19 1513 1266 1418 1266
q20 503 287 322 287
q21 3061 2666 2695 2666
q22 346 277 277 277
Total cold run time: 44841 ms
Total hot run time: 40183 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4399 4395 4427 4395
q2 267 170 179 170
q3 3557 3534 3545 3534
q4 2393 2390 2370 2370
q5 5772 5748 5742 5742
q6 239 121 122 121
q7 2377 1926 1887 1887
q8 3518 3517 3517 3517
q9 9086 9027 9013 9013
q10 3872 3968 3962 3962
q11 501 392 384 384
q12 766 598 598 598
q13 4320 3585 3576 3576
q14 284 263 271 263
q15 571 511 521 511
q16 500 460 470 460
q17 1862 1857 1838 1838
q18 8736 8289 8375 8289
q19 1716 1779 1755 1755
q20 2251 1957 1936 1936
q21 6569 6226 6195 6195
q22 503 433 421 421
Total cold run time: 64059 ms
Total hot run time: 60937 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2005970839
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2072033138
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2069140386
TeamCity be ut coverage result:
Function Coverage: 35.36% (8915/25211)
Line Coverage: 27.10% (73340/270642)
Region Coverage: 26.24% (37882/144366)
Branch Coverage: 23.04% (19291/83716)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c421323f9bc91a5e7ff478b49b125b2507b8fc9d_c421323f9bc91a5e7ff478b49b125b2507b8fc9d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1555644843
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -751,6 +751,16 @@ Version TabletMeta::max_version() const {
return max_version;
}
+size_t TabletMeta::version_count_cross_with_range(const Version& range) const {
+ size_t count = 0;
+ for (const auto& rs_meta : _rs_metas) {
+ if (!(range.first > rs_meta->version().second || range.second < rs_meta->version().first)) {
+ count++;
+ }
+ }
Review Comment:
warning: method 'add_rs_meta' can be made static [readability-convert-member-functions-to-static]
be/src/olap/tablet_meta.h:182:
```diff
- Status add_rs_meta(const RowsetMetaSharedPtr& rs_meta);
+ static Status add_rs_meta(const RowsetMetaSharedPtr& rs_meta);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889628971
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1889596199
run clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1903084420
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1581998882
##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletsProcDir.java:
##########
@@ -52,9 +52,10 @@ public class TabletsProcDir implements ProcDirInterface {
.add("LstSuccessVersion").add("LstFailedVersion").add("LstFailedTime")
.add("LocalDataSize").add("RemoteDataSize").add("RowCount").add("State")
.add("LstConsistencyCheckTime").add("CheckVersion")
- .add("VersionCount").add("QueryHits").add("PathHash").add("Path")
+ .add("VisibleVersionCount").add("TotalVersionCount").add("QueryHits").add("PathHash").add("Path")
Review Comment:
update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #28082:
URL: https://github.com/apache/doris/pull/28082#discussion_r1418874124
##########
be/src/olap/olap_common.h:
##########
@@ -511,4 +513,23 @@ struct RidAndPos {
using PartialUpdateReadPlan = std::map<RowsetId, std::map<uint32_t, std::vector<RidAndPos>>>;
+// used for controll compaction
Review Comment:
control
##########
be/src/olap/tablet.cpp:
##########
@@ -1316,12 +1308,46 @@ std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_single_replica_co
}
std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_base_compaction() {
+ return _pick_visible_rowsets_to_compaction(std::numeric_limits<int64_t>::min(),
+ _cumulative_point - 1);
+}
+
+std::vector<RowsetSharedPtr> Tablet::pick_candidate_rowsets_to_full_compaction() {
+ return pick_candidate_rowsets_to_single_replica_compaction();
+}
+
+std::vector<RowsetSharedPtr> Tablet::_pick_visible_rowsets_to_compaction(
+ int64_t min_start_version, int64_t max_start_version) {
+ int64_t visible_version = std::numeric_limits<int64_t>::max();
+ int32_t keep_invisible_version_limit = config::compaction_keep_invisible_version_max_count;
+ if (auto version_info = std::atomic_load_explicit(&_visible_version, std::memory_order_relaxed);
+ version_info != nullptr) {
+ visible_version = version_info->version.load(std::memory_order_relaxed);
+ bool invisible_timeout = (MonotonicMillis() - version_info->update_ts) >
+ config::compaction_keep_invisible_version_timeout_sec * 1000L;
+ if (invisible_timeout) {
+ keep_invisible_version_limit = config::compaction_keep_invisible_version_min_count;
+ }
+ }
+
std::vector<RowsetSharedPtr> candidate_rowsets;
{
std::shared_lock rlock(_meta_lock);
+ int64_t tablet_max_version = max_version_unlocked().second;
for (const auto& [version, rs] : _rs_version_map) {
- // Do compaction on local rowsets only.
- if (version.first < _cumulative_point && rs->is_local()) {
+ int64_t version_start = version.first;
+ // rowset is remote or rowset is not in given range
+ if (!rs->is_local() || version_start < min_start_version ||
+ version_start > max_start_version) {
+ continue;
+ }
+
+ // can compact, met one of the conditions:
Review Comment:
meet?
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java:
##########
@@ -107,7 +107,7 @@ private static List<List<String>> getTabletStatus(String dbName, String tblName,
row.add(String.valueOf(replica.getLastSuccessVersion()));
row.add(String.valueOf(visibleVersion));
row.add(String.valueOf(replica.getSchemaHash()));
- row.add(String.valueOf(replica.getVersionCount()));
+ row.add(String.valueOf(replica.getTotalVersionCount()));
Review Comment:
visible version count 不用加么？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2076184005
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-2072035773
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](compaction) be do not compact invisible version to avoid query error -230 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28082:
URL: https://github.com/apache/doris/pull/28082#issuecomment-1848582123
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.05 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17219699905 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org