You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2026/04/23 14:47:23 UTC
[PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
mrhhsg opened a new pull request, #62763:
URL: https://github.com/apache/doris/pull/62763
Issue Number: None
Related PR: None
Problem Summary: Add expression-based zone map evaluation to scanners, segment iterators, and Parquet/ORC readers so late-arrival runtime filters and eligible scanner predicates can prune row groups and pages before row materialization. The change also adds BE unit tests and regression coverage for the new pruning path.
Enable expr-based zone map pruning for late runtime filters and eligible Parquet/ORC scanner predicates to reduce unnecessary reads.
- Test: Unit Test
- `./run-be-ut.sh --run scanner_expr_zonemap_test` (the script rebuilt `be/ut_build_ASAN` and launched the BE UT suite in this worktree; the full suite was not awaited to completion before this commit)
- Behavior changed: Yes (late runtime filters and eligible scanner predicates can now participate in expr-based zone map pruning, with new profile counters for the pruning path)
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62763:
URL: https://github.com/apache/doris/pull/62763#issuecomment-4305374301
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62763:
URL: https://github.com/apache/doris/pull/62763#issuecomment-6305435498
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62763:
URL: https://github.com/apache/doris/pull/62763#discussion_r3132122575
##########
be/src/exec/scan/scanner.cpp:
##########
@@ -253,9 +260,58 @@ Status Scanner::try_append_late_arrival_runtime_filter() {
Review Comment:
`_conjuncts` is cleared before we know `clone_conjunct_ctxs()` succeeded. Today the scheduler only logs `try_append_late_arrival_runtime_filter()` failures and keeps scanning, so any clone error here leaves this scanner with an empty `_conjuncts` vector and `_filter_output_block()` stops applying even the original SQL predicates. Please clone into a temporary vector and only swap on success, or propagate the failure so the query stops instead of silently returning unfiltered rows.
##########
be/src/exec/scan/scanner.cpp:
##########
@@ -253,9 +260,58 @@ Status Scanner::try_append_late_arrival_runtime_filter() {
// avoid conjunct destroy in used by storage layer
_conjuncts.clear();
RETURN_IF_ERROR(_local_state->clone_conjunct_ctxs(_conjuncts));
+ if (_late_rf_container != nullptr) {
+ std::lock_guard<std::mutex> lock(_late_rf_container->mutex);
+ _late_rf_container->zone_map_expr_ctxs.clear();
+ for (const auto& conjunct : _conjuncts) {
+ if (_is_zone_map_eligible_expr(conjunct->root().get())) {
+ _late_rf_container->zone_map_expr_ctxs.push_back(conjunct);
+ }
+ }
+ ++_late_rf_container->version;
Review Comment:
`_applied_rf_num` is advanced before `_on_late_arrival_runtime_filter_appended()` succeeds. If the refresh hook fails, the scheduler logs the error and continues, but the next poll now sees `arrived_rf_num == _applied_rf_num` and never retries this late filter. That leaves the reader/storage refresh permanently stale for the rest of the scan. Please move the counter update after the append hook succeeds.
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -521,6 +525,13 @@ Status ParquetReader::_do_init_reader(ReaderInitContext* base_ctx) {
return Status::OK();
}
Review Comment:
This only updates `_lazy_read_ctx.conjuncts`, but the active `RowGroupReader` already copied its `_filter_conjuncts` when it was initialized. If a late runtime filter arrives while the current row group still has unread rows, the current Parquet reader keeps using the old predicate set and misses the new pruning path for that row group. ORC rebuilds and seeks the active reader in its refresh hook; Parquet needs an equivalent refresh here (or some other way to make the current `RowGroupReader` observe the new conjuncts).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #62763:
URL: https://github.com/apache/doris/pull/62763#issuecomment-4305389147
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](be) Enable expr zone map pruning for late runtime filters [doris]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@doris.apache.org>.
github-actions[bot] closed pull request #62763: [improvement](be) Enable expr zone map pruning for late runtime filters
URL: https://github.com/apache/doris/pull/62763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org