You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/10/16 09:08:44 UTC
[PR] [fix](inverted index) Stream compact docs-only SNII postings and align defaults [doris]
airborne12 opened a new pull request, #69192:
URL: https://github.com/apache/doris/pull/69192
### What problem does this PR solve?
Issue Number: CIR-22263
Related PR: None
Problem Summary: SNII streamed compaction rejected docs-only (T1) inverted index postings and rebuilt those indexes from the raw column. The rejection occurred for legacy indexes whose metadata omitted `support_phrase` and for indexes that explicitly disabled phrase positions. This change merges compatible homogeneous T1 postings directly, preserves their physical shape, and keeps mixed or incompatible sources on the rebuild path. It also makes FE persist the documented `support_phrase=true` default and gives BE the same fallback for legacy metadata.
### Release note
Compatible docs-only SNII indexes use streamed compaction. New inverted indexes enable phrase positions by default when `support_phrase` is omitted; explicit `false` remains supported.
### Check List (For Author)
- Test
- [x] Regression test: `test_storage_format_snii` passed on a locally built FE/BE cluster (1 suite).
- [x] Unit Test: 19 focused BE tests and the FE `IndexTest` default test passed.
- [x] Manual test: complete FE and BE builds passed; local FE and BE health checks returned OK.
- Behavior changed:
- [x] Yes. Compatible T1 postings are streamed during compaction, and the omitted phrase setting now defaults to true in FE and BE.
- Does this need documentation?
- [x] No. The documented default already states true.
The complete PR diff passed the English description check, clang-format 16, and Checkstyle on the merged target tree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6218194422
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d_779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d_779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.34% (31694/47772) |
| Line Coverage | 51.09% (333663/653120) |
| Region Coverage | 46.44% (269598/580561) |
| Branch Coverage | 48.17% (126975/263619) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6215611458
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/dbf56734daad6bc136368e2dd0a5cc1cc45f25b7_dbf56734daad6bc136368e2dd0a5cc1cc45f25b7/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/dbf56734daad6bc136368e2dd0a5cc1cc45f25b7_dbf56734daad6bc136368e2dd0a5cc1cc45f25b7/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.35% (31691/47765) |
| Line Coverage | 51.08% (333544/653015) |
| Region Coverage | 46.43% (269524/580458) |
| Branch Coverage | 48.15% (126894/263555) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6214372409
<details>
<summary>TPC-H: <b>Total hot run time: 29700 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbf56734daad6bc136368e2dd0a5cc1cc45f25b7, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17587 5628 5539 5539
q2 2090 297 284 284
q3 10246 1447 836 836
q4 4679 475 349 349
q5 7451 797 534 534
q6 187 187 154 154
q7 785 812 603 603
q8 9244 1318 1048 1048
q9 5589 4320 4289 4289
q10 6820 1302 1032 1032
q11 427 258 234 234
q12 641 402 280 280
q13 18083 3526 3019 3019
q14 276 278 249 249
q15 q16 744 734 682 682
q17 1204 757 584 584
q18 7088 6257 6239 6239
q19 1109 962 608 608
q20 404 350 229 229
q21 5266 2608 2665 2608
q22 395 354 300 300
Total cold run time: 100315 ms
Total hot run time: 29700 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6694 6686 6769 6686
q2 795 617 567 567
q3 5521 5683 5276 5276
q4 2065 2141 1573 1573
q5 5719 5848 5762 5762
q6 256 201 148 148
q7 2281 2019 1809 1809
q8 3366 2749 2756 2749
q9 8112 8016 7885 7885
q10 3638 3591 3235 3235
q11 587 399 371 371
q12 652 689 493 493
q13 3192 3591 3038 3038
q14 287 290 259 259
q15 q16 673 680 622 622
q17 7827 7032 6863 6863
q18 13120 12241 12971 12241
q19 914 790 799 790
q20 2173 2167 1929 1929
q21 5905 4736 4834 4736
q22 505 447 392 392
Total cold run time: 74282 ms
Total hot run time: 67424 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6216370344
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6214063066
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6214525631
<details>
<summary>TPC-DS: <b>Total hot run time: 151650 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbf56734daad6bc136368e2dd0a5cc1cc45f25b7, data reload: false
query5 4300 562 439 439
query6 426 208 195 195
query7 4810 451 223 223
query8 334 165 154 154
query9 8754 3975 3923 3923
query10 462 302 253 253
query11 5991 3789 3529 3529
query12 152 92 87 87
query13 1245 429 317 317
query14 6083 4017 3716 3716
query14_1 3488 3497 3487 3487
query15 210 205 184 184
query16 940 457 417 417
query17 871 661 532 532
query18 2426 415 310 310
query19 203 165 141 141
query20 87 95 84 84
query21 209 131 112 112
query22 12884 12987 12737 12737
query23 13482 12833 12501 12501
query23_1 12636 12591 12512 12512
query24 7096 821 448 448
query24_1 453 467 466 466
query25 516 381 344 344
query26 1305 256 130 130
query27 2768 436 267 267
query28 4635 1908 1913 1908
query29 1556 608 436 436
query30 309 223 182 182
query31 995 929 824 824
query32 139 88 87 87
query33 503 291 229 229
query34 893 839 497 497
query35 743 746 654 654
query36 854 798 745 745
query37 128 103 89 89
query38 1799 1763 1665 1665
query39 800 791 739 739
query39_1 722 728 737 728
query40 217 114 90 90
query41 67 60 61 60
query42 86 78 78 78
query43 372 389 324 324
query44 1276 677 692 677
query45 175 178 159 159
query46 880 967 551 551
query47 3009 2913 2924 2913
query48 287 302 216 216
query49 564 402 298 298
query50 675 286 217 217
query51 10651 10273 10400 10273
query52 78 76 70 70
query53 191 213 150 150
query54 243 194 186 186
query55 70 65 61 61
query56 217 228 213 213
query57 1634 1560 1564 1560
query58 299 258 254 254
query59 2197 2177 1993 1993
query60 285 236 231 231
query61 146 145 144 144
query62 399 337 284 284
query63 189 159 170 159
query64 2721 1020 892 892
query65 3264 3235 3219 3219
query66 1797 437 309 309
query67 20638 20030 20081 20030
query68 2985 993 625 625
query69 381 291 253 253
query70 889 783 769 769
query71 303 229 213 213
query72 2884 2582 2294 2294
query73 512 532 294 294
query74 4570 4454 4280 4280
query75 2352 2314 1920 1920
query76 2298 1019 598 598
query77 351 358 271 271
query78 9781 9702 9155 9155
query79 988 841 530 530
query80 743 449 331 331
query81 538 316 276 276
query82 714 137 111 111
query83 280 201 174 174
query84 316 114 98 98
query85 949 428 369 369
query86 347 239 226 226
query87 2004 1947 1807 1807
query88 3583 2681 2649 2649
query89 346 291 253 253
query90 1762 185 183 183
query91 163 153 120 120
query92 100 81 87 81
query93 1029 1021 577 577
query94 491 313 270 270
query95 579 342 318 318
query96 659 526 235 235
query97 2454 2425 2318 2318
query98 162 156 152 152
query99 786 769 653 653
Total cold run time: 234883 ms
Total hot run time: 151650 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings and align defaults [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6212089741
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6214599941
<details>
<summary>ClickBench: <b>Total hot run time: 24.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dbf56734daad6bc136368e2dd0a5cc1cc45f25b7, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.12 0.11
query4 1.60 0.12 0.12
query5 0.24 0.24 0.22
query6 1.17 0.70 0.68
query7 0.04 0.00 0.00
query8 0.06 0.04 0.03
query9 0.40 0.36 0.32
query10 0.58 0.56 0.60
query11 0.20 0.15 0.14
query12 0.18 0.14 0.15
query13 0.50 0.49 0.49
query14 1.04 1.00 1.02
query15 0.63 0.62 0.62
query16 0.34 0.35 0.36
query17 1.11 1.14 1.15
query18 0.22 0.21 0.21
query19 2.13 1.95 2.03
query20 0.02 0.01 0.01
query21 15.52 0.26 0.14
query22 4.57 0.06 0.06
query23 16.20 0.31 0.12
query24 2.97 0.71 0.37
query25 0.12 0.05 0.04
query26 1.25 0.24 0.16
query27 0.04 0.03 0.04
query28 1.84 0.61 0.38
query29 12.51 4.16 3.32
query30 0.29 0.14 0.13
query31 2.78 0.56 0.34
query32 3.23 0.62 0.51
query33 3.28 3.17 3.17
query34 15.43 4.05 3.40
query35 3.35 3.34 3.35
query36 0.55 0.47 0.47
query37 0.10 0.06 0.06
query38 0.06 0.04 0.03
query39 0.04 0.03 0.03
query40 0.16 0.14 0.13
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.28 s
Total hot run time: 24.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6217003126
<details>
<summary>TPC-DS: <b>Total hot run time: 151829 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d, data reload: false
query5 4317 570 463 463
query6 429 205 192 192
query7 4803 438 230 230
query8 323 170 157 157
query9 8771 3986 3962 3962
query10 464 304 254 254
query11 5952 3772 3537 3537
query12 145 92 87 87
query13 1256 424 332 332
query14 6088 4010 3717 3717
query14_1 3546 3444 3516 3444
query15 205 204 179 179
query16 947 433 415 415
query17 869 657 547 547
query18 2429 420 314 314
query19 200 164 136 136
query20 88 89 84 84
query21 212 131 112 112
query22 12870 12991 12762 12762
query23 13602 12799 12489 12489
query23_1 12621 12540 12628 12540
query24 7123 821 476 476
query24_1 481 469 475 469
query25 537 426 370 370
query26 1268 259 138 138
query27 2772 434 275 275
query28 4610 1961 1964 1961
query29 1554 584 425 425
query30 302 220 194 194
query31 987 920 802 802
query32 160 104 89 89
query33 501 313 235 235
query34 894 857 482 482
query35 727 740 649 649
query36 849 833 726 726
query37 126 100 87 87
query38 1791 1710 1695 1695
query39 799 793 736 736
query39_1 734 733 765 733
query40 217 116 92 92
query41 69 64 65 64
query42 83 80 81 80
query43 360 377 323 323
query44 1293 692 710 692
query45 175 173 159 159
query46 843 964 576 576
query47 2925 2944 2947 2944
query48 292 299 220 220
query49 574 405 289 289
query50 680 274 219 219
query51 10238 10429 10223 10223
query52 79 81 72 72
query53 190 202 152 152
query54 248 189 182 182
query55 71 79 61 61
query56 219 212 237 212
query57 1598 1591 1555 1555
query58 287 265 250 250
query59 2137 2172 1960 1960
query60 326 239 222 222
query61 137 143 147 143
query62 400 340 286 286
query63 194 161 167 161
query64 2724 921 761 761
query65 3298 3233 3215 3215
query66 1758 416 300 300
query67 20177 19980 20285 19980
query68 2980 973 608 608
query69 380 280 246 246
query70 874 779 828 779
query71 298 226 223 223
query72 2782 2498 2328 2328
query73 518 523 303 303
query74 4575 4491 4271 4271
query75 2327 2306 1938 1938
query76 2285 1042 624 624
query77 348 400 288 288
query78 9897 9678 9159 9159
query79 960 852 525 525
query80 1229 456 378 378
query81 575 323 277 277
query82 1020 132 112 112
query83 300 252 179 179
query84 319 122 99 99
query85 858 430 382 382
query86 429 241 226 226
query87 2003 1967 1832 1832
query88 3622 2735 2692 2692
query89 358 295 254 254
query90 1911 181 184 181
query91 162 155 124 124
query92 92 85 89 85
query93 1004 991 572 572
query94 668 301 282 282
query95 595 394 328 328
query96 668 518 234 234
query97 2422 2438 2305 2305
query98 164 152 147 147
query99 768 762 642 642
Total cold run time: 235293 ms
Total hot run time: 151829 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6217096838
<details>
<summary>ClickBench: <b>Total hot run time: 24.36 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d, data reload: false
query1 0.01 0.01 0.01
query2 0.12 0.04 0.04
query3 0.27 0.11 0.11
query4 1.62 0.11 0.11
query5 0.24 0.22 0.25
query6 1.18 0.68 0.68
query7 0.04 0.00 0.00
query8 0.05 0.04 0.04
query9 0.40 0.32 0.36
query10 0.57 0.57 0.58
query11 0.20 0.14 0.14
query12 0.19 0.15 0.14
query13 0.48 0.48 0.49
query14 1.05 1.03 1.00
query15 0.62 0.60 0.60
query16 0.35 0.35 0.34
query17 1.19 1.11 1.15
query18 0.23 0.21 0.21
query19 2.12 1.97 2.01
query20 0.02 0.02 0.01
query21 15.49 0.24 0.15
query22 4.37 0.06 0.05
query23 16.26 0.31 0.13
query24 3.02 0.74 0.44
query25 0.12 0.08 0.05
query26 1.85 0.24 0.15
query27 0.06 0.03 0.03
query28 1.98 0.62 0.40
query29 12.48 4.20 3.30
query30 0.28 0.15 0.13
query31 2.78 0.57 0.34
query32 3.22 0.61 0.53
query33 3.16 3.26 3.13
query34 15.53 4.00 3.36
query35 3.34 3.31 3.37
query36 0.57 0.45 0.41
query37 0.10 0.07 0.06
query38 0.06 0.04 0.04
query39 0.04 0.02 0.02
query40 0.17 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 95.99 s
Total hot run time: 24.36 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6216371134
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6216815614
<details>
<summary>TPC-H: <b>Total hot run time: 29319 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17590 5636 5592 5592
q2 2135 312 256 256
q3 10231 1405 841 841
q4 4687 470 347 347
q5 7444 781 536 536
q6 182 188 156 156
q7 779 804 616 616
q8 9233 1336 1079 1079
q9 5503 4293 4298 4293
q10 6826 1302 1035 1035
q11 427 258 240 240
q12 635 395 278 278
q13 18141 4144 2996 2996
q14 280 278 242 242
q15 q16 729 725 666 666
q17 1213 760 591 591
q18 7185 6232 6148 6148
q19 1115 951 596 596
q20 385 337 236 236
q21 5451 2306 2425 2306
q22 367 327 269 269
Total cold run time: 100538 ms
Total hot run time: 29319 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6146 6154 6143 6143
q2 709 592 559 559
q3 5212 5417 4967 4967
q4 2026 2107 1541 1541
q5 5305 5587 6015 5587
q6 272 197 149 149
q7 2348 2135 1921 1921
q8 3316 3009 2763 2763
q9 8211 8116 8197 8116
q10 3739 3741 3334 3334
q11 624 458 445 445
q12 697 710 517 517
q13 3482 3811 3295 3295
q14 320 326 288 288
q15 q16 689 704 641 641
q17 8374 7559 7381 7381
q18 13098 12277 13101 12277
q19 914 796 786 786
q20 2192 2180 1908 1908
q21 5744 4713 4790 4713
q22 519 461 401 401
Total cold run time: 73937 ms
Total hot run time: 67732 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6219920363
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69192_779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69192_779e1248e6d1bdfb20a0d7fc21f27e1d8b77391d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.21% (35744/46295) |
| Line Coverage | 62.44% (403437/646144) |
| Region Coverage | 58.88% (340600/578473) |
| Branch Coverage | 59.84% (157377/262978) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69192:
URL: https://github.com/apache/doris/pull/69192#discussion_r4294055830
##########
be/src/storage/index/inverted/snii/compaction/eligibility.cpp:
##########
@@ -74,6 +74,22 @@ Status validate_source_shape(const reader::LogicalIndexReader& source, size_t so
return Status::OK();
}
+Status validate_source_shape(const reader::LogicalIndexReader& source, size_t source_ordinal,
+ bool has_positions) {
+ if (source.has_positions() != has_positions ||
+ source.tier() != (has_positions ? format::IndexTier::kT2 : format::IndexTier::kT1)) {
Review Comment:
[P1] Use physical destination IDs before enabling cloud T1 streaming
For a cloud DUP_KEYS table with a docs-only SNII index and `enable_cloud_random_segment_id=true`, accepting T1 here changes compaction from raw rebuild to the streamed owner. `CloudCompactionMixin::construct_output_rowset_writer()` starts output segment IDs above zero, and the index-writer map is keyed by those physical IDs. However, `do_inverted_index_compaction()` selects the SNII destination with `inverted_index_file_writers[destination_ordinal]`; the first lookup at 0 inserts a null writer, which `add_snii_index_streamed()` dereferences and crashes the BE. This docs-only workload previously rebuilt successfully. Resolve the writer using `dest_segment_ids[destination_ordinal]`, as the CLucene path does, and cover nonzero destination IDs before enabling this path for T1.
##########
be/src/storage/index/inverted/snii/compaction/eligibility.cpp:
##########
@@ -238,18 +261,18 @@ Status validate_snii_compaction_eligibility(
}
for (size_t source_ordinal = 0; source_ordinal < sources.size(); ++source_ordinal) {
- if (sources[source_ordinal].reader.get().tier() != format::IndexTier::kT2) {
- return reject(fmt::format("source {} is not a positional (T2) index", source_ordinal));
- }
- RETURN_IF_ERROR(validate_plain_t2_source_eligibility(sources[source_ordinal].reader.get(),
- source_ordinal));
+ RETURN_IF_ERROR(validate_snii_source_eligibility(sources[source_ordinal].reader.get(),
Review Comment:
[P1] Keep gram-family T1 indexes on the rebuild path
A gram analyzer with `support_phrase=false` (or omitted) also produces T1 sources, so this newly accepts more than ordinary docs-only postings. On normal-sized gram segments, the writer deliberately sets `posting_dropped` for common terms. The merge later calls `SniiPostingCursor::prepare_flat_ranges()` -> `LogicalIndexReader::resolve_frq_window()`, which returns `INVERTED_INDEX_EVALUATE_SKIPPED` for those entries. Raw indexing has already been skipped, and the owner does not mark this status for a rebuild on retry, so the same rowsets repeatedly fail compaction. Even sources without dropped postings lose `gram_scheme` in `add_snii_index_streamed()`, disabling subsequent gram index use. Keep gram-family sources on raw rebuild until the merge supports their persisted schemes and dropped postings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #69192:
URL: https://github.com/apache/doris/pull/69192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](inverted index) Stream compact docs-only SNII postings [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #69192:
URL: https://github.com/apache/doris/pull/69192#issuecomment-6214061036
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org