You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "qidaye (via GitHub)" <gi...@apache.org> on 2024/03/12 03:30:19 UTC
[PR] [fix](index compaction)Fix MOW index compaction core [doris]
qidaye opened a new pull request, #32085:
URL: https://github.com/apache/doris/pull/32085
In MOW table with inverted index, when the table has deleted docs, the doc_id in rowid_conversion_map will be INT32_MAX. index_compact in CLucene is not handling it correctly. It will cause doc lost in specific terms and the postings will be incorrect. The index compaction process will be fail.
Here are the changes:
1. Remove INT32_MAX out of destPostingsQueues in CLucene to handle deleted doc right.
2. Add debug code and switch for index compaction in Doris
3. Add debug_index_compaction operation in index_tool
## Proposed changes
Issue Number: close #32084
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #32085:
URL: https://github.com/apache/doris/pull/32085#discussion_r1520806305
##########
be/src/olap/compaction.cpp:
##########
@@ -472,6 +474,45 @@ Status Compaction::do_compaction_impl(int64_t permits) {
dest_index_files[i] = prefix;
}
+ // Only write info files when debug index compaction is enabled.
+ // The files are used to debug index compaction and works with index_tool.
+ if (config::debug_inverted_index_compaction) {
+ auto write_json_to_file = [&](const nlohmann::json& json_obj,
+ const std::string& file_name) {
+ io::FileWriterPtr file_writer;
+ std::string file_path = fmt::format("/tmp/{}.json", file_name);
Review Comment:
use tmp dir in config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990039818
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990175651
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990407175
<details>
<summary>ClickBench: <b>Total hot run time: 29.46 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 883a8d583cbf80c5928bcf68e0a2175ae092a185, data reload: false
query1 0.02 0.01 0.02
query2 0.06 0.02 0.02
query3 0.24 0.05 0.06
query4 1.82 0.11 0.10
query5 0.54 0.52 0.53
query6 1.35 0.62 0.62
query7 0.01 0.00 0.01
query8 0.03 0.02 0.02
query9 0.52 0.47 0.48
query10 0.53 0.53 0.52
query11 0.11 0.09 0.09
query12 0.12 0.09 0.09
query13 0.62 0.64 0.61
query14 0.79 0.78 0.78
query15 0.78 0.75 0.76
query16 0.39 0.39 0.37
query17 1.00 1.03 1.01
query18 0.24 0.24 0.26
query19 1.92 1.88 1.84
query20 0.01 0.01 0.01
query21 15.48 0.55 0.54
query22 2.04 1.85 0.76
query23 17.65 1.14 0.93
query24 5.96 0.73 1.12
query25 0.40 0.12 0.04
query26 0.66 0.16 0.16
query27 0.04 0.04 0.03
query28 7.51 0.73 0.74
query29 12.73 2.33 2.08
query30 0.56 0.51 0.52
query31 2.81 0.39 0.39
query32 3.40 0.50 0.49
query33 3.06 3.04 3.12
query34 15.28 4.81 4.80
query35 4.86 4.85 4.84
query36 1.06 1.01 1.01
query37 0.06 0.04 0.04
query38 0.03 0.02 0.02
query39 0.01 0.02 0.02
query40 0.16 0.14 0.14
query41 0.06 0.01 0.01
query42 0.02 0.02 0.02
query43 0.02 0.02 0.01
Total cold run time: 104.96 s
Total hot run time: 29.46 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990335819
TeamCity be ut coverage result:
Function Coverage: 37.80% (8040/21268)
Line Coverage: 29.48% (65603/222498)
Region Coverage: 28.96% (33762/116588)
Branch Coverage: 24.80% (17324/69854)
Coverage Report: http://coverage.selectdb-in.cc/coverage/883a8d583cbf80c5928bcf68e0a2175ae092a185_883a8d583cbf80c5928bcf68e0a2175ae092a185/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990061516
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990421526
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 883a8d583cbf80c5928bcf68e0a2175ae092a185 with default session variables
Stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 20.0 seconds inserted 10000000 Rows, about 500K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #32085:
URL: https://github.com/apache/doris/pull/32085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990089296
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990067897
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990176834
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990133790
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990187479
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990077062
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990325902
<details>
<summary>TPC-H: <b>Total hot run time: 50381 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 883a8d583cbf80c5928bcf68e0a2175ae092a185, data reload: false
------ Round 1 ----------------------------------
q1 17995 4424 4323 4323
q2 2040 151 151 151
q3 10451 1925 1952 1925
q4 10383 1275 1327 1275
q5 8662 3926 4012 3926
q6 237 123 121 121
q7 2051 1630 1596 1596
q8 9287 2754 2738 2738
q9 11049 10463 10575 10463
q10 8611 3574 3567 3567
q11 430 236 244 236
q12 474 302 305 302
q13 20409 4035 4064 4035
q14 353 333 335 333
q15 504 480 449 449
q16 715 593 595 593
q17 1141 975 990 975
q18 7364 6944 6927 6927
q19 1676 1591 1525 1525
q20 547 299 298 298
q21 4511 4194 4219 4194
q22 550 434 429 429
Total cold run time: 119440 ms
Total hot run time: 50381 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4556 4499 4516 4499
q2 341 246 236 236
q3 4265 4290 4269 4269
q4 2842 2867 2821 2821
q5 7567 7348 7275 7275
q6 241 126 125 125
q7 3257 2806 2812 2806
q8 4340 4524 4499 4499
q9 17118 16809 17098 16809
q10 4226 4260 4266 4260
q11 760 689 688 688
q12 1009 820 851 820
q13 7015 3752 3729 3729
q14 455 430 419 419
q15 492 452 460 452
q16 751 703 693 693
q17 3794 3985 3870 3870
q18 8802 8686 8822 8686
q19 1733 1712 1660 1660
q20 2385 2162 2133 2133
q21 8635 8608 8527 8527
q22 1022 944 923 923
Total cold run time: 85606 ms
Total hot run time: 80199 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990050055
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye commented on code in PR #32085:
URL: https://github.com/apache/doris/pull/32085#discussion_r1520815601
##########
be/src/olap/compaction.cpp:
##########
@@ -472,6 +474,45 @@ Status Compaction::do_compaction_impl(int64_t permits) {
dest_index_files[i] = prefix;
}
+ // Only write info files when debug index compaction is enabled.
+ // The files are used to debug index compaction and works with index_tool.
+ if (config::debug_inverted_index_compaction) {
+ auto write_json_to_file = [&](const nlohmann::json& json_obj,
+ const std::string& file_name) {
+ io::FileWriterPtr file_writer;
+ std::string file_path = fmt::format("/tmp/{}.json", file_name);
Review Comment:
Change it to sys_log_dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](index compaction)Fix MOW index compaction core [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32085:
URL: https://github.com/apache/doris/pull/32085#issuecomment-1990381570
<details>
<summary>TPC-DS: <b>Total hot run time: 202530 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 883a8d583cbf80c5928bcf68e0a2175ae092a185, data reload: false
query1 909 399 381 381
query2 6511 2195 2062 2062
query3 6917 200 199 199
query4 21347 17922 17976 17922
query5 19712 6538 6531 6531
query6 285 210 220 210
query7 4156 300 308 300
query8 253 227 228 227
query9 3175 2726 2629 2629
query10 414 277 297 277
query11 11215 10702 10619 10619
query12 119 71 71 71
query13 5570 642 653 642
query14 17966 13402 13664 13402
query15 351 228 237 228
query16 6430 273 266 266
query17 1525 1471 864 864
query18 2285 412 407 407
query19 207 144 148 144
query20 81 75 78 75
query21 184 92 95 92
query22 5193 5006 5104 5006
query23 32570 32058 32215 32058
query24 7027 6531 6504 6504
query25 517 435 431 431
query26 519 163 161 161
query27 1865 300 308 300
query28 6059 2210 2171 2171
query29 2909 2781 2804 2781
query30 242 162 162 162
query31 900 773 773 773
query32 61 58 66 58
query33 388 254 251 251
query34 865 452 491 452
query35 1099 864 926 864
query36 1727 1387 1645 1387
query37 84 60 59 59
query38 3110 2936 2920 2920
query39 1371 1312 1309 1309
query40 201 104 91 91
query41 36 33 33 33
query42 83 85 90 85
query43 682 636 609 609
query44 1135 703 720 703
query45 244 226 228 226
query46 1235 974 986 974
query47 1895 1713 1724 1713
query48 966 687 665 665
query49 600 361 374 361
query50 871 593 592 592
query51 4733 4756 4768 4756
query52 83 80 76 76
query53 443 319 330 319
query54 2625 2473 2490 2473
query55 83 84 86 84
query56 190 203 213 203
query57 1232 1026 1059 1026
query58 210 204 191 191
query59 3433 3414 3270 3270
query60 206 197 207 197
query61 81 80 79 79
query62 828 516 521 516
query63 483 334 335 334
query64 2457 1463 1415 1415
query65 3617 3549 3564 3549
query66 749 372 359 359
query67 17715 16167 15707 15707
query68 5798 625 636 625
query69 544 355 335 335
query70 1727 1927 1648 1648
query71 408 301 310 301
query72 6365 3421 3428 3421
query73 720 327 327 327
query74 6312 5845 5910 5845
query75 4511 3736 3647 3647
query76 3527 1124 1164 1124
query77 558 244 252 244
query78 12528 11833 11605 11605
query79 10106 637 641 637
query80 1668 391 384 384
query81 490 234 229 229
query82 1711 96 93 93
query83 164 128 130 128
query84 260 67 65 65
query85 868 280 273 273
query86 381 389 384 384
query87 3242 3009 2985 2985
query88 5217 2346 2325 2325
query89 478 285 317 285
query90 1929 196 213 196
query91 149 113 114 113
query92 56 52 48 48
query93 6500 592 560 560
query94 777 204 204 204
query95 1115 1063 1066 1063
query96 656 323 328 323
query97 6390 6313 6372 6313
query98 191 173 178 173
query99 3032 826 880 826
Total cold run time: 311612 ms
Total hot run time: 202530 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org