You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaka11chen (via GitHub)" <gi...@apache.org> on 2026/05/26 07:15:22 UTC
[PR] [fix](ann-index) Fix ANN Range Search State Leakage and Incorrect Slot Index Tracking. [doris]
kaka11chen opened a new pull request, #63666:
URL: https://github.com/apache/doris/pull/63666
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
ANN range search execution state was stored on shared VExpr roots. VExprContext clones share the root expression, so a segment that executed ANN range search could leak that state into another segment without an ANN index and incorrectly remove the common expression. ANN range search also mixed schema column indexes with storage column ids when updating common expression index status, so remapped schemas failed to mark the source slot expression as evaluated. This patch returns ANN execution state through the current evaluation call, stores ANN root bitmap in the current segment IndexContext, and updates slot index status by source column index.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4570030653
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4542168017
<details>
<summary>TPC-H: <b>Total hot run time: 32234 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17854 4149 4070 4070
q2 q3 10833 1457 831 831
q4 4684 488 354 354
q5 7549 2345 2171 2171
q6 236 183 141 141
q7 992 787 649 649
q8 9456 1800 1670 1670
q9 5373 5060 5025 5025
q10 6431 2250 1918 1918
q11 448 267 247 247
q12 637 443 301 301
q13 18184 3410 2797 2797
q14 265 264 236 236
q15 q16 826 776 718 718
q17 965 1065 928 928
q18 7119 5709 5575 5575
q19 1292 1268 1256 1256
q20 550 421 286 286
q21 6293 2967 2750 2750
q22 437 375 311 311
Total cold run time: 100424 ms
Total hot run time: 32234 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4893 4845 4873 4845
q2 q3 4934 5270 4708 4708
q4 2211 2255 1448 1448
q5 4923 4723 4752 4723
q6 233 182 137 137
q7 1875 1763 1601 1601
q8 2521 2179 2279 2179
q9 7526 7471 7522 7471
q10 4807 4749 4276 4276
q11 535 387 366 366
q12 743 758 598 598
q13 3026 3404 2849 2849
q14 272 277 258 258
q15 q16 693 708 624 624
q17 1322 1289 1283 1283
q18 7345 7070 6941 6941
q19 1116 1101 1108 1101
q20 2236 2241 1980 1980
q21 5383 4700 4592 4592
q22 564 483 412 412
Total cold run time: 57158 ms
Total hot run time: 52392 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4552964801
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4562230667
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4570025893
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3309094666
##########
be/src/exprs/vectorized_fn_call.cpp:
##########
@@ -557,7 +557,10 @@ Status VectorizedFnCall::evaluate_ann_range_search(
const std::vector<ColumnId>& idx_to_cid,
const std::vector<std::unique_ptr<segment_v2::ColumnIterator>>& column_iterators,
roaring::Roaring& row_bitmap, segment_v2::AnnIndexStats& ann_index_stats,
- bool enable_result_cache) {
+ bool enable_result_cache, AnnRangeSearchEvaluationResult* evaluation_result) {
+ if (evaluation_result != nullptr) {
Review Comment:
ok
##########
be/src/exprs/vectorized_fn_call.cpp:
##########
@@ -686,17 +690,19 @@ Status VectorizedFnCall::evaluate_ann_range_search(
// If we expected distance but didn't get it, assert in debug to catch logic errors.
DCHECK(!should_have_distance) << "Expected distance from ANN index but got none";
#endif
- _virtual_column_is_fulfilled = false;
}
} else {
// Dest is not virtual column.
- _virtual_column_is_fulfilled = true;
+ dist_fulfilled = true;
}
- _has_been_executed = true;
+ if (evaluation_result != nullptr) {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4553857826
<details>
<summary>TPC-DS: <b>Total hot run time: 172276 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7eafe4602fcde4ab05f1277fb70d12667be19a23, data reload: false
query5 4330 652 513 513
query6 341 219 200 200
query7 4228 590 313 313
query8 342 239 226 226
query9 8818 4119 4095 4095
query10 466 352 305 305
query11 5853 2436 2252 2252
query12 175 125 124 124
query13 1274 609 430 430
query14 6147 5482 5180 5180
query14_1 4484 4599 4547 4547
query15 216 204 190 190
query16 1030 459 423 423
query17 946 722 583 583
query18 2433 510 359 359
query19 223 211 169 169
query20 137 130 132 130
query21 218 145 119 119
query22 13753 13558 13409 13409
query23 17339 16497 16218 16218
query23_1 16534 16350 16349 16349
query24 7447 1762 1353 1353
query24_1 1345 1349 1340 1340
query25 581 500 452 452
query26 1310 325 180 180
query27 2674 579 354 354
query28 4467 1995 2004 1995
query29 1003 658 537 537
query30 308 241 204 204
query31 1146 1086 976 976
query32 90 81 77 77
query33 553 369 312 312
query34 1180 1136 680 680
query35 784 804 723 723
query36 1416 1392 1254 1254
query37 159 108 99 99
query38 3244 3192 3054 3054
query39 929 923 907 907
query39_1 889 895 890 890
query40 243 157 131 131
query41 73 70 69 69
query42 118 116 115 115
query43 349 340 292 292
query44
query45 219 217 198 198
query46 1083 1183 721 721
query47 2410 2405 2278 2278
query48 420 404 304 304
query49 652 516 410 410
query50 1051 367 259 259
query51 4361 4282 4356 4282
query52 110 109 103 103
query53 254 296 214 214
query54 326 301 280 280
query55 98 95 90 90
query56 314 327 330 327
query57 1459 1437 1329 1329
query58 316 314 271 271
query59 1593 1685 1460 1460
query60 329 336 334 334
query61 161 161 159 159
query62 710 650 591 591
query63 241 197 206 197
query64 2436 797 653 653
query65
query66 1720 479 355 355
query67 29719 29722 29411 29411
query68
query69 467 343 308 308
query70 1034 1007 1012 1007
query71 320 277 270 270
query72 3035 2751 2412 2412
query73 829 745 424 424
query74 5140 4921 4810 4810
query75 2684 2614 2269 2269
query76 2328 1137 751 751
query77 404 405 383 383
query78 12424 12586 11788 11788
query79 1481 1016 717 717
query80 641 551 459 459
query81 450 287 243 243
query82 1338 162 126 126
query83 356 284 265 265
query84 308 148 113 113
query85 896 545 461 461
query86 402 360 340 340
query87 3476 3388 3241 3241
query88 3616 2732 2699 2699
query89 457 391 351 351
query90 1942 181 180 180
query91 180 177 145 145
query92 83 84 80 80
query93 1483 1415 901 901
query94 551 329 280 280
query95 675 380 348 348
query96 1030 816 316 316
query97 2762 2759 2660 2660
query98 242 227 239 227
query99 1197 1151 1038 1038
Total cold run time: 254810 ms
Total hot run time: 172276 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3308652459
##########
be/src/exprs/vectorized_fn_call.cpp:
##########
@@ -686,17 +690,19 @@ Status VectorizedFnCall::evaluate_ann_range_search(
// If we expected distance but didn't get it, assert in debug to catch logic errors.
DCHECK(!should_have_distance) << "Expected distance from ANN index but got none";
#endif
- _virtual_column_is_fulfilled = false;
}
} else {
// Dest is not virtual column.
- _virtual_column_is_fulfilled = true;
+ dist_fulfilled = true;
}
- _has_been_executed = true;
+ if (evaluation_result != nullptr) {
Review Comment:
不要写这么多判断，强制外部一定要传一个evaluation_result，代码会简单可靠
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4542292279
<details>
<summary>TPC-DS: <b>Total hot run time: 172815 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c, data reload: false
query5 4320 649 521 521
query6 335 215 214 214
query7 4233 585 306 306
query8 328 233 217 217
query9 8882 4164 4159 4159
query10 451 353 333 333
query11 5826 2587 2329 2329
query12 177 130 124 124
query13 1334 560 437 437
query14 6059 5485 5208 5208
query14_1 4514 4499 4496 4496
query15 212 202 183 183
query16 991 433 418 418
query17 936 708 586 586
query18 2471 506 366 366
query19 217 214 168 168
query20 138 134 140 134
query21 215 140 127 127
query22 13630 13616 13455 13455
query23 17444 16590 16231 16231
query23_1 16501 16347 16394 16347
query24 7447 1809 1336 1336
query24_1 1362 1341 1327 1327
query25 595 509 437 437
query26 1313 319 181 181
query27 2683 594 355 355
query28 4502 2030 2037 2030
query29 997 650 524 524
query30 294 247 201 201
query31 1148 1089 971 971
query32 85 80 76 76
query33 540 368 308 308
query34 1201 1137 667 667
query35 787 791 703 703
query36 1449 1394 1300 1300
query37 158 106 94 94
query38 3240 3188 3070 3070
query39 940 915 895 895
query39_1 879 879 873 873
query40 240 161 130 130
query41 73 69 71 69
query42 119 113 113 113
query43 332 340 309 309
query44
query45 215 209 202 202
query46 1105 1202 756 756
query47 2440 2413 2269 2269
query48 398 401 309 309
query49 651 512 407 407
query50 945 360 264 264
query51 4374 4325 4254 4254
query52 108 107 99 99
query53 262 285 210 210
query54 342 284 265 265
query55 96 93 89 89
query56 301 313 307 307
query57 1436 1441 1338 1338
query58 346 264 261 261
query59 1580 1741 1431 1431
query60 320 312 306 306
query61 159 155 148 148
query62 698 651 586 586
query63 236 199 204 199
query64 2392 796 611 611
query65
query66 1747 480 350 350
query67 29865 29899 29666 29666
query68
query69 464 349 305 305
query70 1075 998 1011 998
query71 308 276 263 263
query72 3065 2732 2448 2448
query73 899 761 426 426
query74 5148 4961 4807 4807
query75 2699 2609 2253 2253
query76 2313 1155 801 801
query77 420 400 370 370
query78 12535 12677 11969 11969
query79 1321 1098 779 779
query80 600 537 458 458
query81 464 286 240 240
query82 242 154 122 122
query83 380 283 243 243
query84 306 140 114 114
query85 966 554 458 458
query86 414 349 313 313
query87 3443 3392 3244 3244
query88 3620 2758 2745 2745
query89 429 406 339 339
query90 1837 186 179 179
query91 183 166 139 139
query92 79 79 73 73
query93 1464 1433 846 846
query94 554 358 296 296
query95 656 391 434 391
query96 1113 811 357 357
query97 2719 2723 2602 2602
query98 235 236 227 227
query99 1175 1142 1044 1044
Total cold run time: 253752 ms
Total hot run time: 172815 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3308451955
##########
be/src/exprs/vectorized_fn_call.cpp:
##########
@@ -557,7 +557,10 @@ Status VectorizedFnCall::evaluate_ann_range_search(
const std::vector<ColumnId>& idx_to_cid,
const std::vector<std::unique_ptr<segment_v2::ColumnIterator>>& column_iterators,
roaring::Roaring& row_bitmap, segment_v2::AnnIndexStats& ann_index_stats,
- bool enable_result_cache) {
+ bool enable_result_cache, AnnRangeSearchEvaluationResult* evaluation_result) {
+ if (evaluation_result != nullptr) {
Review Comment:
为什么是这么写？ 而不是当他== nullptr 的时候直接报错？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4565309297
可以，重新梳理一版，只讲 slot index 这个点。
**1. 两套编号怎么来的**
假设表：
```sql
CREATE TABLE t (
id int,
pad_int int,
pad_text string,
embedding array<float>,
value int,
INDEX idx_embedding(`embedding`) USING ANN ...
);
```
存储层的 `storage cid` 基本来自 tablet schema 的列顺序，写 segment 时会写进 column meta：
```text
storage cid:
0 -> id
1 -> pad_int
2 -> pad_text
3 -> embedding
4 -> value
```
而 `schema index` 是当前这次 scan 的 reader schema / block 里的列位置。它不是固定表列编号，而是“当前读出来的第几列”。
比如查询：
```sql
SELECT id
FROM t
WHERE l2_distance_approximate(embedding, [0,0,0]) < 1.0;
```
这次只需要 `id` 和 `embedding`：
```text
reader schema index: 0 1
reader column: id embedding
storage cid: 0 3
```
所以当前 `_schema->column_ids()` 是：
```cpp
idx_to_cid = [0, 3]
```
表达式 rebind 之后，`embedding` 的 `SlotRef::column_id()` 会被改成当前 reader schema 位置：
```text
embedding slot_ref->column_id() = 1
```
ANN prepare 时保存的就是这个值：
```cpp
range_search_runtime.src_col_idx = slot_ref->column_id(); // 1
```
所以这里：
```text
src_col_idx = 1 // schema index / block index
idx_to_cid[src_col_idx] = 3 // storage cid
```
**2. 旧代码为什么 index-only scan 没 work**
ANN range search 本身查 index 时是对的：
```cpp
src_col_cid = idx_to_cid[src_col_idx]; // idx_to_cid[1] = 3
```
也就是说它能正确用 `embedding` 的 ANN index 过滤 row bitmap。
问题发生在过滤完以后，Doris 想标记：
```text
embedding 这个 common expr 已经被 index 处理过了。
如果 embedding 不输出，只用于这个 ANN predicate，就不用再读 embedding 数据列。
```
状态 map 大概长这样：
```cpp
_common_expr_index_exec_status[3][expr] = false;
_common_expr_to_slotref_map[ctx][1] = expr;
```
注意：
```text
_common_expr_index_exec_status 用 storage cid: 3
_common_expr_to_slotref_map 用 schema index: 1
```
旧代码却这样查：
```cpp
auto src_col_idx = 1;
ColumnId cid = idx_to_cid[src_col_idx]; // 3
if (slot_ref_map.find(cid) == slot_ref_map.end()) {
return Status::OK();
}
```
它拿 `3` 去查 `slot_ref_map`，但 `slot_ref_map` 的 key 是 `1`，所以查不到，直接 return。后面的状态标记没有执行。
于是状态仍然是：
```cpp
_common_expr_index_exec_status[3][expr] = false;
```
后续 `_check_all_conditions_passed_inverted_index_for_column(3, ...)` 会认为 `embedding` 上的 expr 没有完全被 index 覆盖，因此不能跳过读取 `embedding` 数据列。
结果就是：
```text
ANN index 已经过滤了 row bitmap，
但 embedding 还是会被读出来再计算表达式。
```
所以之前不是结果错，而是 **index-only scan 优化没有生效**。
正确修法是保持坐标系一致：
```cpp
src_col_idx = 1;
slot_ref_map.find(src_col_idx); // 用 schema index 找 expr
set_true_for_index_status(expr, src_col_idx); // 也传 schema index
```
`set_true_for_index_status` 内部再做：
```cpp
column_id = _col_ids[src_col_idx]; // _col_ids[1] = 3
_expr_index_status[3][expr] = true;
```
最终状态变成：
```cpp
_common_expr_index_exec_status[3][expr] = true;
```
这样当查询只是：
```sql
SELECT id
FROM t
WHERE l2_distance_approximate(embedding, [0,0,0]) < 1.0;
```
且 `embedding` 不需要输出时，ANN index 已经满足 predicate，BE 就可以不再读取 `embedding` 数据列，index-only scan 才真正 work。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3310434930
##########
be/src/exprs/vexpr_context.cpp:
##########
@@ -440,41 +441,57 @@ Status VExprContext::evaluate_ann_range_search(
const std::unordered_map<VExprContext*, std::unordered_map<ColumnId, VExpr*>>&
common_expr_to_slotref_map,
roaring::Roaring& row_bitmap, segment_v2::AnnIndexStats& ann_index_stats,
- bool enable_result_cache) {
+ bool enable_result_cache, bool* ann_range_search_executed) {
+ if (ann_range_search_executed != nullptr) {
+ *ann_range_search_executed = false;
+ }
if (_root == nullptr) {
return Status::OK();
}
+ AnnRangeSearchEvaluationResult evaluation_result;
RETURN_IF_ERROR(_root->evaluate_ann_range_search(
_ann_range_search_runtime, cid_to_index_iterators, idx_to_cid, column_iterators,
- row_bitmap, ann_index_stats, enable_result_cache));
+ row_bitmap, ann_index_stats, enable_result_cache, evaluation_result));
- if (!_root->ann_range_search_executedd()) {
+ if (!evaluation_result.executed) {
return Status::OK();
}
+ if (ann_range_search_executed != nullptr) {
+ *ann_range_search_executed = true;
+ }
+
+ DCHECK(_index_context != nullptr);
+ _index_context->set_index_result_for_expr(
+ _root.get(),
+ segment_v2::InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(row_bitmap),
+ std::make_shared<roaring::Roaring>()));
- if (!_root->ann_dist_is_fulfilled()) {
+ if (!evaluation_result.dist_fulfilled) {
// Do not perform index scan in this case.
return Status::OK();
}
- auto src_col_idx = _ann_range_search_runtime.src_col_idx;
+ DCHECK_LT(_ann_range_search_runtime.src_col_idx, idx_to_cid.size());
+ const auto src_col_idx = cast_set<int>(_ann_range_search_runtime.src_col_idx);
Review Comment:
为啥同一个变量，一个cast 成 colidx 一个就是col key 呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3310430092
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1386,14 +1390,6 @@ Status SegmentIterator::_apply_index_expr() {
_opts.stats->ann_index_range_cache_hits += ann_index_stats.range_cache_hits.value();
}
- for (auto it = _common_expr_ctxs_push_down.begin(); it != _common_expr_ctxs_push_down.end();) {
Review Comment:
这边统一 erase 了。
```
// Always apply expr-level index (e.g., search expressions) if we have common_expr_pushdown
// This allows search expressions with variant subcolumns to be evaluated even when
// the segment doesn't have all subcolumns
RETURN_IF_ERROR(_apply_index_expr());
for (auto it = _common_expr_ctxs_push_down.begin();
it != _common_expr_ctxs_push_down.end();) {
if ((*it)->all_expr_inverted_index_evaluated()) {
const auto* result = (*it)->get_index_context()->get_index_result_for_expr(
(*it)->root().get());
if (result != nullptr) {
_row_bitmap &= *result->get_data_bitmap();
it = _common_expr_ctxs_push_down.erase(it);
}
} else {
++it;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN Range Search State Leakage and Incorrect Slot Index Tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4541520413
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4543533257
# BE Regression && UT Coverage Report
Increment line coverage `95.74% (45/47)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63666_ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63666_ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.75% (28037/38018) |
| Line Coverage | 57.64% (304228/527814) |
| Region Coverage | 54.79% (254543/464592) |
| Branch Coverage | 56.27% (109909/195333) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4554870848
# BE Regression && UT Coverage Report
Increment line coverage `95.45% (42/44)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63666_7eafe4602fcde4ab05f1277fb70d12667be19a23_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63666_7eafe4602fcde4ab05f1277fb70d12667be19a23_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.79% (28058/38025) |
| Line Coverage | 57.69% (304516/527864) |
| Region Coverage | 54.75% (254455/464767) |
| Branch Coverage | 56.28% (109940/195352) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4551759761
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4552983637
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #63666:
URL: https://github.com/apache/doris/pull/63666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN Range Search State Leakage and Incorrect Slot Index Tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4541521327
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4553780443
<details>
<summary>TPC-H: <b>Total hot run time: 32161 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7eafe4602fcde4ab05f1277fb70d12667be19a23, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17644 4129 4144 4129
q2 q3 10820 1454 824 824
q4 4685 484 345 345
q5 7659 2284 2142 2142
q6 247 179 138 138
q7 972 783 662 662
q8 9374 1823 1661 1661
q9 5277 5076 5000 5000
q10 6399 2218 1918 1918
q11 459 278 254 254
q12 635 430 300 300
q13 18109 3341 2789 2789
q14 268 269 245 245
q15 q16 816 762 714 714
q17 993 966 1025 966
q18 7085 5738 5513 5513
q19 1237 1271 1171 1171
q20 574 445 303 303
q21 6141 2876 2773 2773
q22 557 376 314 314
Total cold run time: 99951 ms
Total hot run time: 32161 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4865 4894 4774 4774
q2 q3 4969 5310 4661 4661
q4 2131 2228 1411 1411
q5 5046 4651 4807 4651
q6 233 178 126 126
q7 1888 1760 1557 1557
q8 2433 2189 2168 2168
q9 7950 7490 7432 7432
q10 4756 4665 4184 4184
q11 540 392 387 387
q12 738 740 522 522
q13 3065 3350 2831 2831
q14 268 280 269 269
q15 q16 680 701 619 619
q17 1308 1281 1274 1274
q18 7289 6831 6875 6831
q19 1132 1094 1132 1094
q20 2229 2206 1944 1944
q21 5304 4614 4499 4499
q22 516 445 395 395
Total cold run time: 57340 ms
Total hot run time: 51629 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3310392355
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1386,14 +1390,6 @@ Status SegmentIterator::_apply_index_expr() {
_opts.stats->ann_index_range_cache_hits += ann_index_stats.range_cache_hits.value();
}
- for (auto it = _common_expr_ctxs_push_down.begin(); it != _common_expr_ctxs_push_down.end();) {
Review Comment:
如果这里没有erase了，那么在哪里将这个expr erase 了呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on code in PR #63666:
URL: https://github.com/apache/doris/pull/63666#discussion_r3310590911
##########
be/src/exprs/vexpr_context.cpp:
##########
@@ -440,41 +441,57 @@ Status VExprContext::evaluate_ann_range_search(
const std::unordered_map<VExprContext*, std::unordered_map<ColumnId, VExpr*>>&
common_expr_to_slotref_map,
roaring::Roaring& row_bitmap, segment_v2::AnnIndexStats& ann_index_stats,
- bool enable_result_cache) {
+ bool enable_result_cache, bool* ann_range_search_executed) {
+ if (ann_range_search_executed != nullptr) {
+ *ann_range_search_executed = false;
+ }
if (_root == nullptr) {
return Status::OK();
}
+ AnnRangeSearchEvaluationResult evaluation_result;
RETURN_IF_ERROR(_root->evaluate_ann_range_search(
_ann_range_search_runtime, cid_to_index_iterators, idx_to_cid, column_iterators,
- row_bitmap, ann_index_stats, enable_result_cache));
+ row_bitmap, ann_index_stats, enable_result_cache, evaluation_result));
- if (!_root->ann_range_search_executedd()) {
+ if (!evaluation_result.executed) {
return Status::OK();
}
+ if (ann_range_search_executed != nullptr) {
+ *ann_range_search_executed = true;
+ }
+
+ DCHECK(_index_context != nullptr);
+ _index_context->set_index_result_for_expr(
+ _root.get(),
+ segment_v2::InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(row_bitmap),
+ std::make_shared<roaring::Roaring>()));
- if (!_root->ann_dist_is_fulfilled()) {
+ if (!evaluation_result.dist_fulfilled) {
// Do not perform index scan in this case.
return Status::OK();
}
- auto src_col_idx = _ann_range_search_runtime.src_col_idx;
+ DCHECK_LT(_ann_range_search_runtime.src_col_idx, idx_to_cid.size());
+ const auto src_col_idx = cast_set<int>(_ann_range_search_runtime.src_col_idx);
Review Comment:
因为有两个地方要调用，需要不同类型的参数。
`slot_ref_map.find(src_col_key)`
`_index_context->set_true_for_index_status(slot_ref_expr_addr, src_col_idx)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #63666:
URL: https://github.com/apache/doris/pull/63666#issuecomment-4565527352
refactor the test of index only scan by using debug point.
https://github.com/apache/doris/pull/63859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org