You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rgbuilds (via GitHub)" <gi...@apache.org> on 2026/09/28 03:13:30 UTC
[PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
rgbuilds opened a new pull request, #25822:
URL: https://github.com/apache/datafusion/pull/25822
## Which issue does this PR close?
- Closes #18355.
## Rationale for this change
`row_groups_pruned_bloom_filter` currently reports retained row groups as Bloom filter matches even when Bloom pruning was not evaluated—for example, when Bloom filters are disabled or unavailable, no relevant Bloom statistics were loaded, or predicate evaluation failed.
This makes `EXPLAIN ANALYZE` suggest that Bloom filters evaluated and matched row groups when they did not participate in the pruning decision. It also displays an idle Bloom pruning metric when all its counters are zero.
A retained row group is not necessarily a Bloom filter match. The metric should describe actual Bloom pruning outcomes.
## What changes are included in this PR?
- Record a Bloom filter match only when the Bloom pruning predicate is successfully evaluated and determines that the row group may match.
- Leave Bloom pruning counters unchanged when Bloom pruning is unavailable or cannot be evaluated.
- Continue retaining row groups conservatively when Bloom predicate evaluation fails, while recording the failure in `predicate_evaluation_errors`.
- Omit `row_groups_pruned_bloom_filter` from physical-plan displays when all of its pruning counters are zero.
- Preserve the registered metric and direct metric lookup; only its accounting and displayed output change.
- Update affected Rust assertions and SQL logic test snapshots.
The row-group access decisions and query results are unchanged.
## What is the testing strategy for this PR?
Added `bloom_filter_pruning_error_retains_row_group_without_match`, which verifies that a Bloom predicate evaluation error:
- retains the row group;
- increments `predicate_evaluation_errors`;
- does not increment either the Bloom matched or pruned counter.
The display tests verify that:
- an idle Bloom pruning metric is omitted from aggregated and full Indent, Graphviz, and PostgreSQL JSON output;
- unrelated zero-valued metrics remain visible;
- genuine nonzero Bloom pruning results remain visible.
The following existing test coverage was also run:
- Parquet Bloom-filter and row-group-filter unit tests;
- `parquet_integration` row-group-pruning tests;
- `core_integration` explain-analyze tests;
- physical-plan display tests;
- affected SQL logic tests for dynamic filtering, dynamic row-group pruning, explain-analyze, limit pruning, and Parquet filter pushdown.
Formatting checks also pass.
## Are there any user-facing changes?
Yes. Parquet Bloom filter pruning metrics in `EXPLAIN ANALYZE` and other physical-plan displays more accurately represent actual Bloom filter evaluation. An entirely idle `row_groups_pruned_bloom_filter` metric is no longer displayed.
There are no public API changes and no changes to query results or row-group access decisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-6027090500
@xudong963 I rebased onto current main and fixed the merge-queue failure in dynamic_row_group_pruning.slt. The relevant tests and extended suite pass locally. The new upstream workflow runs are awaiting approval again — could you please approve them when convenient? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5921915741
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.1.0 (current)
Built [ 61.209s] (current)
Parsing datafusion v55.1.0 (current)
Parsed [ 0.035s] (current)
Building datafusion v55.1.0 (baseline)
Built [ 62.895s] (baseline)
Parsing datafusion v55.1.0 (baseline)
Parsed [ 0.037s] (baseline)
Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.637s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 126.600s] datafusion
Building datafusion-datasource-parquet v55.1.0 (current)
Built [ 54.222s] (current)
Parsing datafusion-datasource-parquet v55.1.0 (current)
Parsed [ 0.034s] (current)
Building datafusion-datasource-parquet v55.1.0 (baseline)
Built [ 51.320s] (baseline)
Parsing datafusion-datasource-parquet v55.1.0 (baseline)
Parsed [ 0.036s] (baseline)
Checking datafusion-datasource-parquet v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.159s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 106.984s] datafusion-datasource-parquet
Building datafusion-physical-plan v55.1.0 (current)
Built [ 39.431s] (current)
Parsing datafusion-physical-plan v55.1.0 (current)
Parsed [ 0.172s] (current)
Building datafusion-physical-plan v55.1.0 (baseline)
Built [ 39.530s] (baseline)
Parsing datafusion-physical-plan v55.1.0 (baseline)
Parsed [ 0.177s] (baseline)
Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.646s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 81.345s] datafusion-physical-plan
Building datafusion-sqllogictest v55.1.0 (current)
Built [ 103.551s] (current)
Parsing datafusion-sqllogictest v55.1.0 (current)
Parsed [ 0.022s] (current)
Building datafusion-sqllogictest v55.1.0 (baseline)
Built [ 102.876s] (baseline)
Parsing datafusion-sqllogictest v55.1.0 (baseline)
Parsed [ 0.024s] (baseline)
Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.091s] 223 checks: 221 pass, 2 fail, 0 warn, 31 skip
--- failure function_missing: pub fn removed or renamed ---
Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_missing.ron
Failed in:
function datafusion_sqllogictest::memory_drift_tracker, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/da347d550619edfbf1f1f395b86615d66cf9f7f7/datafusion/sqllogictest/src/memory_drift.rs:145
function datafusion_sqllogictest::enable_memory_drift_logging, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/da347d550619edfbf1f1f395b86615d66cf9f7f7/datafusion/sqllogictest/src/memory_drift.rs:135
function datafusion_sqllogictest::flush_thread_allocations, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/da347d550619edfbf1f1f395b86615d66cf9f7f7/datafusion/sqllogictest/src/memory_drift.rs:87
--- failure struct_missing: pub struct removed or renamed ---
Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/struct_missing.ron
Failed in:
struct datafusion_sqllogictest::CountingAllocator, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/da347d550619edfbf1f1f395b86615d66cf9f7f7/datafusion/sqllogictest/src/memory_drift.rs:56
Summary semver requires new major version: 2 major and 0 minor checks failed
Finished [ 209.632s] datafusion-sqllogictest
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5964047541
> > @xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?
>
>
>
> Plan to leave it for two days to see if others wanna have a look
Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5956451686
@xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 merged PR #25822:
URL: https://github.com/apache/datafusion/pull/25822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5922108861
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25822?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.50704%` with `33 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.52%. Comparing base ([`1bd2ad3`](https://app.codecov.io/gh/apache/datafusion/commit/1bd2ad3efe70829c9524c5218aff8b4f2a93edbe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a99b6c7`](https://app.codecov.io/gh/apache/datafusion/commit/a99b6c70e645d72b9c454b9861aa0a4a92ef3d3f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25822?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/display.rs](https://app.codecov.io/gh/apache/datafusion/pull/25822?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fdisplay.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9kaXNwbGF5LnJz) | 81.46% | [33 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25822?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25822 +/- ##
========================================
Coverage 82.51% 82.52%
========================================
Files 1141 1141
Lines 439774 439965 +191
Branches 439774 439965 +191
========================================
+ Hits 362896 363069 +173
- Misses 54939 54947 +8
- Partials 21939 21949 +10
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25822?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-6052415852
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5936368252
@xudong963 Thanks for approving the earlier workflow runs. I’ve since updated the Bloom accounting for the fully matched path introduced by #25854 and added regression tests. Fork CI is green on 22cf2a2c7; the upstream workflows are awaiting approval again. Could you please approve the latest runs when convenient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5924023163
Rebased onto current main and updated the accounting for the fully matched row-group path introduced by #25854.
Extended the existing scan test to verify Bloom matched/pruned counters for statistics-disabled, mixed, and entirely fully matched cases. Restoring the previous increment makes the mixed-case assertion fail.
Focused Bloom, row-group, display, and affected SQL logic tests pass locally, along with formatting checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: report Parquet Bloom filter pruning metrics accurately [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #25822:
URL: https://github.com/apache/datafusion/pull/25822#issuecomment-5964035575
> @xudong963 Thanks for reviewing and approving! Could you please merge this when convenient?
Plan to leave it for two days to see if others wanna have a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org