You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/24 23:34:54 UTC
[PR] feat: keep optional filters pruning-only in the Parquet post-scan path [datafusion]
adriangb opened a new pull request, #25722:
URL: https://github.com/apache/datafusion/pull/25722
# feat: keep optional filters pruning-only in the Parquet post-scan path
## Which issue does this PR close?
- Part of #22883. Design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt
- Follow-up of #22384. **Depends on #22384 and #25673.** Review only the top commit.
- Fixes the 1.16x TPC-H regression that the benchmark bot found on #22384, when #25681 marks the dynamic filters as optional.
```mermaid
graph LR
FS["filter_stats commit"]
A1["#25673 Optional wrapper"]
A2["#25681 producers mark filters"]
A3["#25674 gate"]
A4["#25682 Parquet consumer"]
B1["#25683 FilterExec consumer"]
B2["FilterExec reordering (new)"]
C2["#25713 split join filter"]
P["#22384 post-scan filter"]
F["post-scan skips optional filters (new)"]
FS --> A3
FS --> B2
A1 --> A2
C2 --> A2
A1 --> A4
A3 --> A4
A1 --> B1
A3 --> B1
A1 --> F
P --> F
classDef this fill:#f6e7d6,stroke:#b25e12,stroke-width:3px
class F this
```
## Rationale for this change
With `pushdown_filters = false`, #22384 makes the scan evaluate all accepted filters for each row after the decode. This includes the hash join dynamic filter. The join checks the same rows again, so this work gives nothing, and it caused a 1.16x TPC-H regression on #22384.
Optional filters (#25673) are not needed for correctness. The post-scan filter must not evaluate them.
## What changes are included in this PR?
The scan splits the predicate with `split_optional` (#25673):
| Case | Required conjunct | Optional conjunct |
|---|---|---|
| `pushdown_filters = false` | post-scan filter | statistics, page index, bloom filter and file pruning only (as before #22384) |
| `pushdown_filters = true`, row filter accepts it | row filter | row filter |
| `pushdown_filters = true`, row filter rejects it for a file | post-scan filter | not used for that file |
| row filter build error for the whole file | post-scan filter | not used for that file |
```diff
// pushdown_filters = false
- post_scan = split_conjunction(predicate)
+ post_scan = split_optional(predicate).0 // required conjuncts only
```
The change is about 10 lines in `opener/mod.rs`, `push_decoder.rs` and `row_filter.rs`. It does not need the gate or `optional_filter_mode` (#25674, #25682).
## What is the testing strategy for this PR?
`optional_conjunct_is_never_evaluated_post_scan` (opener unit test) reads a file with each predicate and counts the rows that the post-scan filter sees (`post_scan_rows_pruned + post_scan_rows_matched`):
| Predicate | `pushdown_filters` | Output rows | Post-scan rows |
|---|---|---|---|
| `s IS NOT NULL` (row filter rejects it) | false | 2 | 3 |
| `s IS NOT NULL` | true | 2 | 3 |
| `id > 1` | false | 2 | 3 |
| `Optional(s IS NOT NULL)` | false | 3 | 0 |
| `Optional(s IS NOT NULL)` | true | 3 | 0 |
| `Optional(id > 1)` | false | 3 | 0 |
| `Optional(id > 1)` (row filter accepts it) | true | 2 | 0 |
`datasource-parquet` lib tests, `parquet_integration`, `core_integration` and the full sqllogictest suite pass.
## Are there any user-facing changes?
No. Without optional filters (before #25681) nothing changes. With them, `pushdown_filters = false` scans do the same row-level work as before #22384 for dynamic filters.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: keep optional filters pruning-only in the Parquet post-scan path [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25722:
URL: https://github.com/apache/datafusion/pull/25722#issuecomment-5824315922
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25722?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `86.36364%` with `129 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.50%. Comparing base ([`1be6b04`](https://app.codecov.io/gh/apache/datafusion/commit/1be6b04c9dd3a910258d2da828fd9526cfccf0b1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d197b69`](https://app.codecov.io/gh/apache/datafusion/commit/d197b690cb9c838acb121bfcd77eb9b3b75fa69a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25722?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/proto-models/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fproto-models%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1tb2RlbHMvc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [57 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/physical-expr/src/expressions/optional\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fexpressions%2Foptional_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9leHByZXNzaW9ucy9vcHRpb25hbF9maWx0ZXIucnM=) | 79.53% | [17 Missing and 18 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...usion/datasource-parquet/src/decoder\_projection.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fdecoder_projection.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL2RlY29kZXJfcHJvamVjdGlvbi5ycw==) | 93.36% | [3 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/push\_decoder.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fpush_decoder.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3B1c2hfZGVjb2Rlci5ycw==) | 81.69% | [7 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/row\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Frow_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3Jvd19maWx0ZXIucnM=) | 95.74% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/utils/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Futils%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy91dGlscy9tb2QucnM=) | 96.47% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/simplifier/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fsimplifier%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9zaW1wbGlmaWVyL21vZC5ycw==) | 88.88% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/physical\_plan/from\_proto.rs](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Fphysical_plan%2Ffrom_proto.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvcGh5c2ljYWxfcGxhbi9mcm9tX3Byb3RvLnJz) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25722 +/- ##
=========================================
Coverage 82.49% 82.50%
=========================================
Files 1140 1141 +1
Lines 438781 439862 +1081
Branches 438781 439862 +1081
=========================================
+ Hits 361965 362894 +929
- Misses 54938 55022 +84
- Partials 21878 21946 +68
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25722?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: keep optional filters pruning-only in the Parquet post-scan path [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25722:
URL: https://github.com/apache/datafusion/pull/25722#issuecomment-5824178642
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.1.0 (current)
Built [ 59.941s] (current)
Parsing datafusion v55.1.0 (current)
Parsed [ 0.034s] (current)
Building datafusion v55.1.0 (baseline)
Built [ 56.019s] (baseline)
Parsing datafusion v55.1.0 (baseline)
Parsed [ 0.034s] (baseline)
Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.828s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 118.861s] datafusion
Building datafusion-datasource-parquet v55.1.0 (current)
Built [ 47.839s] (current)
Parsing datafusion-datasource-parquet v55.1.0 (current)
Parsed [ 0.035s] (current)
Building datafusion-datasource-parquet v55.1.0 (baseline)
Built [ 48.008s] (baseline)
Parsing datafusion-datasource-parquet v55.1.0 (baseline)
Parsed [ 0.035s] (baseline)
Checking datafusion-datasource-parquet v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.207s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ParquetFileMetrics.post_scan_rows_pruned in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/metrics.rs:100
field ParquetFileMetrics.post_scan_rows_matched in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/metrics.rs:102
field ParquetFileMetrics.post_scan_filter_eval_time in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/metrics.rs:104
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 97.218s] datafusion-datasource-parquet
Building datafusion-physical-expr v55.1.0 (current)
Built [ 28.072s] (current)
Parsing datafusion-physical-expr v55.1.0 (current)
Parsed [ 0.048s] (current)
Building datafusion-physical-expr v55.1.0 (baseline)
Built [ 28.000s] (baseline)
Parsing datafusion-physical-expr v55.1.0 (baseline)
Parsed [ 0.048s] (baseline)
Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.440s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 57.636s] datafusion-physical-expr
Building datafusion-proto v55.1.0 (current)
Built [ 52.376s] (current)
Parsing datafusion-proto v55.1.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-proto v55.1.0 (baseline)
Built [ 52.796s] (baseline)
Parsing datafusion-proto v55.1.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.154s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 106.924s] datafusion-proto
Building datafusion-proto-models v55.1.0 (current)
Built [ 23.605s] (current)
Parsing datafusion-proto-models v55.1.0 (current)
Parsed [ 0.129s] (current)
Building datafusion-proto-models v55.1.0 (baseline)
Built [ 24.189s] (baseline)
Parsing datafusion-proto-models v55.1.0 (baseline)
Parsed [ 0.131s] (baseline)
Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 2.636s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant ExprType:OptionalFilter in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1686
variant ExprType:OptionalFilter in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1686
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 51.790s] datafusion-proto-models
Building datafusion-pruning v55.1.0 (current)
Built [ 38.882s] (current)
Parsing datafusion-pruning v55.1.0 (current)
Parsed [ 0.016s] (current)
Building datafusion-pruning v55.1.0 (baseline)
Built [ 39.115s] (baseline)
Parsing datafusion-pruning v55.1.0 (baseline)
Parsed [ 0.016s] (baseline)
Checking datafusion-pruning v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.105s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 79.048s] datafusion-pruning
Building datafusion-sqllogictest v55.1.0 (current)
Built [ 94.402s] (current)
Parsing datafusion-sqllogictest v55.1.0 (current)
Parsed [ 0.056s] (current)
Building datafusion-sqllogictest v55.1.0 (baseline)
Built [ 94.167s] (baseline)
Parsing datafusion-sqllogictest v55.1.0 (baseline)
Parsed [ 0.023s] (baseline)
Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.120s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 191.322s] datafusion-sqllogictest
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org