You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/24 04:31:44 UTC
[PR] feat: let the Parquet scan skip non-selective optional filters [datafusion]
adriangb opened a new pull request, #25682:
URL: https://github.com/apache/datafusion/pull/25682
## Which issue does this PR close?
- Part of #22883 (experimental "optional filters" stack, design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt). Related to #20324, #3463, #19858.
- **Depends on #25673 and #25674.** Review only the top commit. Works without the producer PR, but real queries only produce optional filters once that PR lands.
## Rationale for this change
With `datafusion.execution.parquet.pushdown_filters = true`, the Parquet scan evaluates pushed-down dynamic filters row by row. When a dynamic filter removes few rows (for example a hash join where most probe keys match), we evaluate it and then the join checks the same rows again. This is a main cause of the regressions that keep `pushdown_filters` off by default (#20324, #19858).
## What changes are included in this PR?
The Parquet scan uses `datafusion.execution.optional_filter_mode` (from #25674) for optional conjuncts (found with `split_optional`, root AND chain only). Required conjuncts do not change.
| mode | optional conjuncts |
|---|---|
| `always` (default) | exactly today's behavior |
| `pruning_only` | not added to the `RowFilter`; still used for file, row group, page and bloom filter pruning |
| `adaptive` | each optional conjunct gets its own `RowFilter` predicate behind an `OptionalFilterGate`. A paused gate returns an all-true mask without evaluating. |
Details:
- Optional predicates always run after the required ones, so the gate measures only what the optional filter adds.
- Pooled gate statistics live on the `ParquetSource` and are shared by all partitions and files. A dynamic filter is keyed by `expression_id()`, other optional filters by position. A new file starts from the pooled verdict.
- A gate lives for one file and survives per-row-group row filter rebuilds. The dynamic filter inside `Optional` stays live, so the gate resets when the filter changes.
- An optional conjunct that the row filter rejects for a file (for example after schema evolution) is dropped for that file. This is safe because it is optional (compare #25268).
- New metrics: `optional_filter_rows_skipped`, `optional_filter_pauses`.
- When `pushdown_filters = false` nothing changes.
## What is the testing strategy for this PR?
- Unit tests for candidate building per mode, ordering, dropped rejected conjuncts, pausing with exact row counts, pooled seeding, site keys, and config plumbing.
- Integration tests (`core/tests/parquet/optional_filters.rs`, 10 row groups × 2000 rows): identical results in all modes, metrics per mode, a selective filter never pauses, and a dynamic filter updated while its gate is paused takes effect at the next row group.
- `optional_filters.slt`: join and TopK queries give the same results in all three modes. This file will show the new metrics once the producer PR lands.
Benchmarks (TPC-H, TPC-DS, ClickBench with `pushdown_filters = true` and each mode) will follow once the producer PR is stacked in.
## Are there any user-facing changes?
New behavior only when `optional_filter_mode` is set to `adaptive` or `pruning_only`. The default keeps today's behavior. Two new scan metrics. Known limitation: the mode is not serialized to protobuf (a decoded `ParquetSource` uses `always`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: let the Parquet scan skip non-selective optional filters [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25682:
URL: https://github.com/apache/datafusion/pull/25682#issuecomment-5807956388
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25682?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.59061%` with `129 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.52%. Comparing base ([`95bb0a0`](https://app.codecov.io/gh/apache/datafusion/commit/95bb0a0dfa48ca20d017faa55b2fc96ee2f300fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4004f26`](https://app.codecov.io/gh/apache/datafusion/commit/4004f26d06bb78b0948cfa9708d831a05d71dfcd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25682?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/proto-models/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fproto-models%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1tb2RlbHMvc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [57 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/physical-expr/src/expressions/optional\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fexpressions%2Foptional_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9leHByZXNzaW9ucy9vcHRpb25hbF9maWx0ZXIucnM=) | 79.53% | [17 Missing and 18 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/row\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Frow_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3Jvd19maWx0ZXIucnM=) | 97.96% | [4 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/source.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fsource.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3NvdXJjZS5ycw==) | 85.96% | [7 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...tafusion/physical-expr/src/optional\_filter\_gate.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Foptional_filter_gate.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9vcHRpb25hbF9maWx0ZXJfZ2F0ZS5ycw==) | 98.28% | [6 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/utils/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Futils%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy91dGlscy9tb2QucnM=) | 96.94% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/metrics.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fmetrics.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL21ldHJpY3MucnM=) | 92.85% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/common/src/config.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fcommon%2Fsrc%2Fconfig.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL2NvbmZpZy5ycw==) | 91.30% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/simplifier/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fsimplifier%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9zaW1wbGlmaWVyL21vZC5ycw==) | 88.88% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/physical\_plan/from\_proto.rs](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Fphysical_plan%2Ffrom_proto.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvcGh5c2ljYWxfcGxhbi9mcm9tX3Byb3RvLnJz) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25682?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25682 +/- ##
==========================================
+ Coverage 82.48% 82.52% +0.03%
==========================================
Files 1140 1143 +3
Lines 437614 439116 +1502
Branches 437614 439116 +1502
==========================================
+ Hits 360986 362379 +1393
- Misses 54834 54911 +77
- Partials 21794 21826 +32
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25682?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: let the Parquet scan skip non-selective optional filters [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25682:
URL: https://github.com/apache/datafusion/pull/25682#issuecomment-5807831710
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.1.0 (current)
Built [ 46.736s] (current)
Parsing datafusion v55.1.0 (current)
Parsed [ 0.029s] (current)
Building datafusion v55.1.0 (baseline)
Built [ 46.396s] (baseline)
Parsing datafusion v55.1.0 (baseline)
Parsed [ 0.029s] (baseline)
Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.744s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 96.546s] datafusion
Building datafusion-common v55.1.0 (current)
Built [ 25.825s] (current)
Parsing datafusion-common v55.1.0 (current)
Parsed [ 0.054s] (current)
Building datafusion-common v55.1.0 (baseline)
Built [ 26.255s] (baseline)
Parsing datafusion-common v55.1.0 (baseline)
Parsed [ 0.054s] (baseline)
Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.846s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ExecutionOptions.optional_filter_mode in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:949
field ExecutionOptions.optional_filter_max_pass_ratio in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:949
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 54.282s] datafusion-common
Building datafusion-datasource-parquet v55.1.0 (current)
Built [ 40.777s] (current)
Parsing datafusion-datasource-parquet v55.1.0 (current)
Parsed [ 0.031s] (current)
Building datafusion-datasource-parquet v55.1.0 (baseline)
Built [ 40.961s] (baseline)
Parsing datafusion-datasource-parquet v55.1.0 (baseline)
Parsed [ 0.030s] (baseline)
Checking datafusion-datasource-parquet v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.181s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 83.246s] datafusion-datasource-parquet
Building datafusion-physical-expr v55.1.0 (current)
Built [ 23.516s] (current)
Parsing datafusion-physical-expr v55.1.0 (current)
Parsed [ 0.043s] (current)
Building datafusion-physical-expr v55.1.0 (baseline)
Built [ 24.287s] (baseline)
Parsing datafusion-physical-expr v55.1.0 (baseline)
Parsed [ 0.042s] (baseline)
Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.452s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 49.395s] datafusion-physical-expr
Building datafusion-proto v55.1.0 (current)
Built [ 45.575s] (current)
Parsing datafusion-proto v55.1.0 (current)
Parsed [ 0.015s] (current)
Building datafusion-proto v55.1.0 (baseline)
Built [ 44.928s] (baseline)
Parsing datafusion-proto v55.1.0 (baseline)
Parsed [ 0.017s] (baseline)
Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.148s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 91.987s] datafusion-proto
Building datafusion-proto-models v55.1.0 (current)
Built [ 19.959s] (current)
Parsing datafusion-proto-models v55.1.0 (current)
Parsed [ 0.111s] (current)
Building datafusion-proto-models v55.1.0 (baseline)
Built [ 20.024s] (baseline)
Parsing datafusion-proto-models v55.1.0 (baseline)
Parsed [ 0.113s] (baseline)
Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 2.291s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant ExprType:OptionalFilter in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1686
variant ExprType:OptionalFilter in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1686
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 43.791s] datafusion-proto-models
Building datafusion-pruning v55.1.0 (current)
Built [ 32.893s] (current)
Parsing datafusion-pruning v55.1.0 (current)
Parsed [ 0.014s] (current)
Building datafusion-pruning v55.1.0 (baseline)
Built [ 33.057s] (baseline)
Parsing datafusion-pruning v55.1.0 (baseline)
Parsed [ 0.014s] (baseline)
Checking datafusion-pruning v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.096s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 67.324s] datafusion-pruning
Building datafusion-sqllogictest v55.1.0 (current)
Built [ 79.129s] (current)
Parsing datafusion-sqllogictest v55.1.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-sqllogictest v55.1.0 (baseline)
Built [ 79.266s] (baseline)
Parsing datafusion-sqllogictest v55.1.0 (baseline)
Parsed [ 0.020s] (baseline)
Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.115s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 161.747s] datafusion-sqllogictest
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org