You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2024/01/09 07:42:26 UTC
[PR] [Enhancement](index tool) refine inverted index tool code [doris]
airborne12 opened a new pull request, #29717:
URL: https://github.com/apache/doris/pull/29717
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882720570
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882573566
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882600334
<details>
<summary>TPC-H: <b>Total hot run time: 39311 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bac5553f10454f91625bc5577c6a91f302410a33, data reload: false
------ Round 1 ----------------------------------
q1 18331 5538 5563 5538
q2 2594 167 156 156
q3 11289 1097 1097 1097
q4 10251 814 853 814
q5 7888 2976 2938 2938
q6 204 132 130 130
q7 917 519 511 511
q8 9311 1975 2004 1975
q9 6486 6387 6357 6357
q10 8228 3061 2965 2965
q11 432 234 217 217
q12 386 230 240 230
q13 18274 3467 3431 3431
q14 240 209 217 209
q15 554 518 502 502
q16 458 403 412 403
q17 934 765 696 696
q18 7337 6658 6649 6649
q19 1552 1515 1484 1484
q20 722 324 301 301
q21 2734 2414 2370 2370
q22 381 338 340 338
Total cold run time: 109503 ms
Total hot run time: 39311 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5521 5533 5509 5509
q2 345 262 269 262
q3 3322 3331 3346 3331
q4 2118 2072 2055 2055
q5 5838 5844 5858 5844
q6 203 123 124 123
q7 2289 1878 1894 1878
q8 3480 3574 3540 3540
q9 8887 8942 8885 8885
q10 3848 3870 3851 3851
q11 599 517 499 499
q12 801 628 651 628
q13 6379 3229 3230 3229
q14 302 258 257 257
q15 560 520 503 503
q16 555 487 493 487
q17 2416 2400 2390 2390
q18 8664 8576 8554 8554
q19 1570 1616 1613 1613
q20 2194 1986 1967 1967
q21 5576 5419 5340 5340
q22 574 549 516 516
Total cold run time: 66041 ms
Total hot run time: 61261 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29717:
URL: https://github.com/apache/doris/pull/29717#discussion_r1445739423
##########
be/src/index-tools/index_tool.cpp:
##########
@@ -187,8 +187,15 @@ int main(int argc, char** argv) {
std::string dir_str = p.parent_path().string();
std::string file_str = p.filename().string();
auto fs = doris::io::global_local_filesystem();
+ bool is_exists = false;
+ const auto* file_path = dir_str + "/" + file_str;
Review Comment:
warning: 'auto file_path' can be declared as 'const auto *file_path' [readability-qualified-auto]
```suggestion
const auto *file_path = dir_str + "/" + file_str;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #29717:
URL: https://github.com/apache/doris/pull/29717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882616507
<details>
<summary>TPC-DS: <b>Total hot run time: 187976 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bac5553f10454f91625bc5577c6a91f302410a33, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 984 374 363 363
query2 6880 2089 2075 2075
query3 6718 237 226 226
query4 25593 23016 22705 22705
query5 6146 628 620 620
query6 340 232 217 217
query7 4636 313 304 304
query8 257 241 235 235
query9 9732 3342 3305 3305
query10 703 359 332 332
query11 16564 15493 15755 15493
query12 174 128 117 117
query13 1651 345 342 342
query14 12425 7700 7657 7657
query15 256 245 249 245
query16 6381 353 351 351
query17 1741 529 550 529
query18 1990 349 338 338
query19 264 171 181 171
query20 126 113 123 113
query21 190 107 98 98
query22 4839 4564 4529 4529
query23 32720 31623 31785 31623
query24 12522 2969 3003 2969
query25 619 407 388 388
query26 1780 189 201 189
query27 2975 327 322 322
query28 7124 2131 2126 2126
query29 2142 427 435 427
query30 301 158 155 155
query31 1039 868 862 862
query32 135 108 100 100
query33 841 360 374 360
query34 916 499 507 499
query35 994 909 887 887
query36 1433 1320 1335 1320
query37 229 106 106 106
query38 3541 3383 3354 3354
query39 1542 1336 1325 1325
query40 310 104 96 96
query41 43 43 37 37
query42 123 111 117 111
query43 617 572 587 572
query44 1222 846 838 838
query45 218 208 205 205
query46 1110 743 722 722
query47 1799 1671 1684 1671
query48 376 310 293 293
query49 1225 340 341 340
query50 739 370 377 370
query51 5560 5404 5324 5324
query52 135 105 106 105
query53 237 177 177 177
query54 1409 720 747 720
query55 122 108 106 106
query56 280 310 307 307
query57 1060 1015 1011 1011
query58 367 350 310 310
query59 3436 3128 3156 3128
query60 383 321 305 305
query61 129 131 152 131
query62 589 526 488 488
query63 227 210 209 209
query64 6010 1824 1852 1824
query65 3611 3455 3462 3455
query66 1334 403 435 403
query67 16134 15886 15501 15501
query68 12511 565 571 565
query69 641 389 372 372
query70 1894 1762 1731 1731
query71 604 315 296 296
query72 5460 3443 3499 3443
query73 2893 338 339 338
query74 7096 6767 6604 6604
query75 5353 2398 2368 2368
query76 6434 999 1240 999
query77 763 373 388 373
query78 9552 9274 8859 8859
query79 1082 536 562 536
query80 567 380 366 366
query81 482 227 226 226
query82 243 134 124 124
query83 193 149 154 149
query84 256 58 58 58
query85 953 303 273 273
query86 447 450 461 450
query87 3651 3526 3445 3445
query88 3213 2542 2546 2542
query89 392 309 345 309
query90 1942 282 278 278
query91 134 98 100 98
query92 108 99 97 97
query93 1162 433 494 433
query94 890 257 262 257
query95 581 522 529 522
query96 641 345 348 345
query97 4487 4262 4273 4262
query98 259 238 225 225
query99 1193 897 919 897
Total cold run time: 304325 ms
Total hot run time: 187976 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882720675
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882574855
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882599962
TeamCity be ut coverage result:
Function Coverage: 36.64% (8640/23580)
Line Coverage: 28.65% (70208/245052)
Region Coverage: 27.60% (36316/131570)
Branch Coverage: 24.29% (18554/76396)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bac5553f10454f91625bc5577c6a91f302410a33_bac5553f10454f91625bc5577c6a91f302410a33/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882566121
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29717:
URL: https://github.com/apache/doris/pull/29717#discussion_r1445748186
##########
be/src/index-tools/index_tool.cpp:
##########
@@ -187,8 +187,15 @@ int main(int argc, char** argv) {
std::string dir_str = p.parent_path().string();
std::string file_str = p.filename().string();
auto fs = doris::io::global_local_filesystem();
+ bool is_exists = false;
+ const auto file_path = dir_str + "/" + file_str;
Review Comment:
warning: 'const auto file_path' can be declared as 'const auto *const file_path' [readability-qualified-auto]
```suggestion
const auto *const file_path = dir_str + "/" + file_str;
```
##########
be/src/index-tools/index_tool.cpp:
##########
@@ -211,8 +222,15 @@
std::string dir_str = p.parent_path().string();
std::string file_str = p.filename().string();
auto fs = doris::io::global_local_filesystem();
+ bool is_exists = false;
+ const auto file_path = dir_str + "/" + file_str;
Review Comment:
warning: 'const auto file_path' can be declared as 'const auto *const file_path' [readability-qualified-auto]
```suggestion
const auto *const file_path = dir_str + "/" + file_str;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882569735
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882616603
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.94 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.4 seconds inserted 10000000 Rows, about 364K ops/s
storage size: 17183830965 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29717:
URL: https://github.com/apache/doris/pull/29717#discussion_r1445732221
##########
be/src/index-tools/index_tool.cpp:
##########
@@ -211,8 +222,15 @@
std::string dir_str = p.parent_path().string();
std::string file_str = p.filename().string();
auto fs = doris::io::global_local_filesystem();
+ bool is_exists = false;
+ auto file_path = dir_str + "/" + file_str;
Review Comment:
warning: 'auto file_path' can be declared as 'const auto *file_path' [readability-qualified-auto]
```suggestion
const auto *file_path = dir_str + "/" + file_str;
```
##########
be/src/index-tools/index_tool.cpp:
##########
@@ -187,8 +187,15 @@ int main(int argc, char** argv) {
std::string dir_str = p.parent_path().string();
std::string file_str = p.filename().string();
auto fs = doris::io::global_local_filesystem();
+ bool is_exists = false;
+ auto file_path = dir_str + "/" + file_str;
Review Comment:
warning: 'auto file_path' can be declared as 'const auto *file_path' [readability-qualified-auto]
```suggestion
const auto *file_path = dir_str + "/" + file_str;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882552620
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](index tool) refine inverted index tool code [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #29717:
URL: https://github.com/apache/doris/pull/29717#issuecomment-1882560924
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org