You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2026/04/16 11:08:35 UTC
[PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Mryange opened a new pull request, #62561:
URL: https://github.com/apache/doris/pull/62561
## What problem does this PR solve?
Previously, `_process_common_expr` in `SegmentIterator` had a workaround that temporarily replaced column 0 with a const default column of `selected_size` rows. This was necessary because `execute_conjuncts_and_filter_block` internally used `block->rows()` to determine the filter size, but during lazy materialization the block columns could have different row counts than the actual `selected_size`.
Now that `VExpr::execute_filter` and `VExpr::execute_column` accept an explicit `count`/`num_rows` parameter, this mock column trick is no longer needed. The filter size can be driven by `selected_size` directly.
## Changes
1. **Removed mock column logic from `_process_common_expr`** — deleted the `need_mock_col` / `col0` save/restore code and the associated comment.
2. **Added new `execute_conjuncts_and_filter_block` overload with explicit `num_rows`** — a new 6-argument overload in `VExprContext` that uses `num_rows` instead of `block->rows()` for the filter size. Internally it calls `execute_filter` with the explicit row count.
3. **Deleted old 5-argument overload** — the previous `execute_conjuncts_and_filter_block(ctxs, block, columns_to_filter, column_to_keep, filter)` overload (which used `block->rows()`) was removed. Its only external caller (`table_function_operator.cpp`) was migrated to the new 6-arg overload with `output_block->rows()`.
4. **Replaced `block->rows()` with `select_size` in `_output_index_result_column`** — the early return check, `vec_match_pred.resize`, `null_map_vec.resize`, and DCHECK all now use the `select_size` parameter instead of `block->rows()`.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4417119150
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange closed pull request #62561: [refine](storage) remove mock column workaround in segment iterator common expr evaluation
URL: https://github.com/apache/doris/pull/62561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4259535102
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4265818403
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4417492436
<details>
<summary>TPC-H: <b>Total hot run time: 29508 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9aba2f81d1fbcc85e75b4b2b63aefbd25043a728, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17626 3853 3845 3845
q2 q3 10713 890 615 615
q4 4660 460 363 363
q5 7640 1346 1143 1143
q6 284 177 137 137
q7 951 947 790 790
q8 10630 1417 1284 1284
q9 6277 5386 5331 5331
q10 6338 2083 1797 1797
q11 474 261 255 255
q12 685 417 290 290
q13 18240 3343 2723 2723
q14 293 287 262 262
q15 q16 913 875 791 791
q17 1014 1035 701 701
q18 6551 5696 5625 5625
q19 1232 1228 1102 1102
q20 497 391 264 264
q21 4544 2273 1886 1886
q22 419 352 304 304
Total cold run time: 99981 ms
Total hot run time: 29508 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4170 4071 4060 4060
q2 q3 4620 4801 4175 4175
q4 2076 2148 1390 1390
q5 4966 4988 5269 4988
q6 189 166 130 130
q7 2030 2274 1652 1652
q8 3474 3154 3183 3154
q9 8524 8430 8385 8385
q10 4534 4520 4253 4253
q11 659 427 391 391
q12 712 776 539 539
q13 3262 3640 2925 2925
q14 305 311 279 279
q15 q16 798 791 707 707
q17 1375 1306 1406 1306
q18 8122 7056 7159 7056
q19 1158 1171 1157 1157
q20 2279 2256 1956 1956
q21 6149 5365 4928 4928
q22 556 506 449 449
Total cold run time: 59958 ms
Total hot run time: 53880 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4259535244
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4417969194
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (20/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62561_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62561_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.60% (27781/37746) |
| Line Coverage | 57.49% (300362/522504) |
| Region Coverage | 54.61% (249964/457684) |
| Branch Coverage | 56.22% (108165/192403) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62561:
URL: https://github.com/apache/doris/pull/62561#discussion_r3092830874
##########
be/src/exprs/vexpr_context.h:
##########
@@ -314,9 +314,13 @@ class VExprContext {
const VExprContextSPtrs& ctxs, Block* block, std::vector<uint32_t>& columns_to_filter,
int column_to_keep);
+ // Uses an explicit num_rows instead of block->rows() to determine the filter size.
+ // This is needed when block columns may have different row counts
+ // (e.g., during segment iterator lazy materialization).
static Status execute_conjuncts_and_filter_block(const VExprContextSPtrs& ctxs, Block* block,
std::vector<uint32_t>& columns_to_filter,
- int column_to_keep, IColumn::Filter& filter);
+ int column_to_keep, IColumn::Filter& filter,
+ size_t num_rows);
Review Comment:
This signature change removes the previous 5-arg overload, but `be/src/exec/operator/table_function_operator.cpp:718` still calls `execute_conjuncts_and_filter_block(..., filter)` without the new `num_rows` argument. With only the 4-arg and 6-arg overloads left, this PR no longer builds. Please either update that caller (likely with `output_block->rows()`) or keep a wrapper overload here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4263923384
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4261034510
# BE UT Coverage Report
Increment line coverage `65.00% (13/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.16% (20204/38008) |
| Line Coverage | 36.71% (190254/518288) |
| Region Coverage | 32.94% (147641/448221) |
| Branch Coverage | 34.09% (64662/189698) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4417527485
<details>
<summary>TPC-DS: <b>Total hot run time: 172017 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9aba2f81d1fbcc85e75b4b2b63aefbd25043a728, data reload: false
query5 4326 654 518 518
query6 334 230 210 210
query7 4222 564 322 322
query8 328 232 214 214
query9 8821 4049 4029 4029
query10 474 372 303 303
query11 5821 2442 2210 2210
query12 187 128 126 126
query13 1293 632 449 449
query14 6592 5415 5048 5048
query14_1 4344 4374 4324 4324
query15 210 210 192 192
query16 1005 467 430 430
query17 1127 744 601 601
query18 2707 489 352 352
query19 216 203 164 164
query20 137 134 133 133
query21 216 143 115 115
query22 13679 13637 13385 13385
query23 17210 16385 16903 16385
query23_1 16379 16352 16326 16326
query24 7789 1807 1424 1424
query24_1 1423 1390 1383 1383
query25 573 542 460 460
query26 1640 335 179 179
query27 2948 583 356 356
query28 4450 2006 1971 1971
query29 1021 641 537 537
query30 320 240 206 206
query31 1336 1087 944 944
query32 96 73 73 73
query33 573 329 289 289
query34 1160 1133 620 620
query35 757 780 661 661
query36 1363 1350 1154 1154
query37 148 102 84 84
query38 3217 3143 3037 3037
query39 927 931 903 903
query39_1 861 871 858 858
query40 248 157 132 132
query41 63 60 59 59
query42 111 107 109 107
query43 323 324 285 285
query44
query45 212 204 190 190
query46 1065 1239 715 715
query47 2361 2308 2191 2191
query48 408 443 298 298
query49 626 505 427 427
query50 694 290 223 223
query51 4368 4361 4226 4226
query52 105 104 92 92
query53 256 276 199 199
query54 311 279 250 250
query55 93 86 83 83
query56 304 310 289 289
query57 1433 1427 1293 1293
query58 312 276 272 272
query59 1591 1645 1435 1435
query60 344 337 322 322
query61 159 157 153 153
query62 680 616 559 559
query63 252 200 204 200
query64 2402 891 729 729
query65
query66 1707 531 400 400
query67 29965 30032 29907 29907
query68
query69 458 337 305 305
query70 1040 1002 965 965
query71 307 271 270 270
query72 2929 2734 2496 2496
query73 843 812 429 429
query74 5104 4899 4758 4758
query75 2817 2634 2337 2337
query76 2294 1111 746 746
query77 413 423 341 341
query78 12889 12946 12305 12305
query79 1504 1000 746 746
query80 669 573 485 485
query81 457 289 240 240
query82 1270 168 126 126
query83 354 275 250 250
query84 254 142 113 113
query85 868 519 437 437
query86 400 334 311 311
query87 3429 3384 3226 3226
query88 3550 2664 2632 2632
query89 448 400 340 340
query90 1878 187 182 182
query91 193 188 155 155
query92 84 75 78 75
query93 959 966 564 564
query94 552 365 322 322
query95 681 482 360 360
query96 1064 795 334 334
query97 2692 2707 2574 2574
query98 244 232 231 231
query99 1147 1136 970 970
Total cold run time: 255333 ms
Total hot run time: 172017 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4259739877
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4259970929
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](storage) remove mock column workaround in segment iterator common expr evaluation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62561:
URL: https://github.com/apache/doris/pull/62561#issuecomment-4261241905
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (20/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62561_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62561_9aba2f81d1fbcc85e75b4b2b63aefbd25043a728_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.71% (27433/37217) |
| Line Coverage | 57.35% (296310/516670) |
| Region Coverage | 54.53% (246655/452331) |
| Branch Coverage | 56.19% (106919/190266) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org