You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/06 12:52:05 UTC
(datafusion) branch gh-readonly-queue/main/pr-25822-720c5dfba5e7e3bb402d02e300cb9fa485a1f674 created (now 440bfd2e10)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-25822-720c5dfba5e7e3bb402d02e300cb9fa485a1f674
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 440bfd2e10 fix: report Parquet Bloom filter pruning metrics accurately (#25822)
This branch includes the following new commits:
new 440bfd2e10 fix: report Parquet Bloom filter pruning metrics accurately (#25822)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: report Parquet Bloom filter pruning metrics accurately (#25822)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25822-720c5dfba5e7e3bb402d02e300cb9fa485a1f674
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 440bfd2e10421f09bc558cf0120650a2f18d55ae
Author: Rishabh Gupta <ri...@gmail.com>
AuthorDate: Tue Oct 6 12:51:43 2026 +0000
fix: report Parquet Bloom filter pruning metrics accurately (#25822)
## Which issue does this PR close?
- Closes #18355.
## Rationale for this change
`row_groups_pruned_bloom_filter` currently reports retained row groups
as Bloom filter matches even when Bloom pruning was not evaluated—for
example, when Bloom filters are disabled or unavailable, no relevant
Bloom statistics were loaded, or predicate evaluation failed.
This makes `EXPLAIN ANALYZE` suggest that Bloom filters evaluated and
matched row groups when they did not participate in the pruning
decision. It also displays an idle Bloom pruning metric when all its
counters are zero.
A retained row group is not necessarily a Bloom filter match. The metric
should describe actual Bloom pruning outcomes.
## What changes are included in this PR?
- Record a Bloom filter match only when the Bloom pruning predicate is
successfully evaluated and determines that the row group may match.
- Leave Bloom pruning counters unchanged when Bloom pruning is
unavailable or cannot be evaluated.
- Continue retaining row groups conservatively when Bloom predicate
evaluation fails, while recording the failure in
`predicate_evaluation_errors`.
- Omit `row_groups_pruned_bloom_filter` from physical-plan displays when
all of its pruning counters are zero.
- Preserve the registered metric and direct metric lookup; only its
accounting and displayed output change.
- Update affected Rust assertions and SQL logic test snapshots.
The row-group access decisions and query results are unchanged.
## What is the testing strategy for this PR?
Added `bloom_filter_pruning_error_retains_row_group_without_match`,
which verifies that a Bloom predicate evaluation error:
- retains the row group;
- increments `predicate_evaluation_errors`;
- does not increment either the Bloom matched or pruned counter.
The display tests verify that:
- an idle Bloom pruning metric is omitted from aggregated and full
Indent, Graphviz, and PostgreSQL JSON output;
- unrelated zero-valued metrics remain visible;
- genuine nonzero Bloom pruning results remain visible.
The following existing test coverage was also run:
- Parquet Bloom-filter and row-group-filter unit tests;
- `parquet_integration` row-group-pruning tests;
- `core_integration` explain-analyze tests;
- physical-plan display tests;
- affected SQL logic tests for dynamic filtering, dynamic row-group
pruning, explain-analyze, limit pruning, and Parquet filter pushdown.
Formatting checks also pass.
## Are there any user-facing changes?
Yes. Parquet Bloom filter pruning metrics in `EXPLAIN ANALYZE` and other
physical-plan displays more accurately represent actual Bloom filter
evaluation. An entirely idle `row_groups_pruned_bloom_filter` metric is
no longer displayed.
There are no public API changes and no changes to query results or
row-group access decisions.
---
datafusion/core/tests/parquet/row_group_pruning.rs | 106 +++--
datafusion/core/tests/sql/explain_analyze.rs | 15 +-
datafusion/datasource-parquet/src/opener/mod.rs | 54 ++-
.../datasource-parquet/src/row_group_filter.rs | 57 ++-
datafusion/physical-plan/src/display.rs | 502 +++++++++++++++++----
.../test_files/dynamic_filter_pushdown_config.slt | 2 +-
.../test_files/dynamic_row_group_pruning.slt | 4 +-
.../sqllogictest/test_files/explain_analyze.slt | 20 +-
.../sqllogictest/test_files/limit_pruning.slt | 4 +-
.../test_files/push_down_filter_parquet.slt | 68 +--
10 files changed, 616 insertions(+), 216 deletions(-)
diff --git a/datafusion/core/tests/parquet/row_group_pruning.rs b/datafusion/core/tests/parquet/row_group_pruning.rs
index 0721715921..c15df47f62 100644
--- a/datafusion/core/tests/parquet/row_group_pruning.rs
+++ b/datafusion/core/tests/parquet/row_group_pruning.rs
@@ -245,7 +245,7 @@ async fn prune_timestamps_nanos() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(10)
.test_row_group_prune()
@@ -263,7 +263,7 @@ async fn prune_timestamps_micros() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(10)
.test_row_group_prune()
@@ -281,7 +281,7 @@ async fn prune_timestamps_millis() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(10)
.test_row_group_prune()
@@ -299,7 +299,7 @@ async fn prune_timestamps_seconds() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(10)
.test_row_group_prune()
@@ -315,7 +315,7 @@ async fn prune_date32() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(3))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(1)
.test_row_group_prune()
@@ -359,7 +359,7 @@ async fn prune_disabled() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(10)
.test_row_group_prune()
@@ -384,7 +384,7 @@ async fn prune_disabled() {
// This should not prune any
assert_eq!(output.predicate_evaluation_errors(), Some(0));
- assert_eq!(output.row_groups_matched(), Some(4));
+ assert_eq!(output.row_groups_matched(), Some(0));
assert_eq!(output.row_groups_pruned(), Some(0));
assert_eq!(
output.result_rows,
@@ -409,7 +409,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(11)
.test_row_group_prune()
@@ -424,7 +424,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(11)
.test_row_group_prune()
@@ -474,7 +474,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(3)
.test_row_group_prune()
@@ -490,7 +490,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(2)
.test_row_group_prune()
@@ -506,7 +506,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(9)
.test_row_group_prune()
@@ -551,6 +551,8 @@ macro_rules! int_tests {
#[tokio::test]
async fn $fn_eq_in_list_negated() {
// result of sql "SELECT * FROM t where not in (1)" prune nothing
+ // Groups without 1 are fully matched by statistics and skip Bloom.
+ // Only [0, 5) contains 1, so only that group is Bloom-evaluated.
RowGroupPruningTest::new()
.with_scenario(Scenario::Int)
.with_query(&format!("SELECT * FROM t where i{} not in (1)", $bits))
@@ -558,7 +560,7 @@ macro_rules! int_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(1))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(19)
.test_row_group_prune()
@@ -608,7 +610,7 @@ macro_rules! uint_tests {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(11)
.test_row_group_prune()
@@ -658,7 +660,7 @@ macro_rules! uint_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(2)
.test_row_group_prune()
@@ -674,7 +676,7 @@ macro_rules! uint_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(2)
.test_row_group_prune()
@@ -718,7 +720,9 @@ macro_rules! uint_tests {
#[tokio::test]
async fn $fn_eq_in_list_negated() {
- // result of sql "SELECT * FROM t where not in (1)" prune nothing
+ // result of sql "SELECT * FROM t where not in (6)" prune nothing
+ // Groups without 6 are fully matched by statistics and skip Bloom.
+ // Only [5, 10) contains 6, so only that group is Bloom-evaluated.
RowGroupPruningTest::new()
.with_scenario(Scenario::UInt)
.with_query(&format!("SELECT * FROM t where u{} not in (6)", $bits))
@@ -726,7 +730,7 @@ macro_rules! uint_tests {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(1))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(19)
.test_row_group_prune()
@@ -814,7 +818,7 @@ async fn prune_f64_lt() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(11)
.test_row_group_prune()
@@ -826,7 +830,7 @@ async fn prune_f64_lt() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(11)
.test_row_group_prune()
@@ -844,7 +848,7 @@ async fn prune_f64_scalar_fun_and_gt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(2))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(1)
.test_row_group_prune()
@@ -861,7 +865,7 @@ async fn prune_f64_scalar_fun() {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(1)
.test_row_group_prune()
@@ -878,7 +882,7 @@ async fn prune_f64_complex_expr() {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(9)
.test_row_group_prune()
@@ -895,7 +899,7 @@ async fn prune_f64_complex_expr_subtract() {
.with_matched_by_stats(Some(4))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(4))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(9)
.test_row_group_prune()
@@ -914,7 +918,7 @@ async fn prune_decimal_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(6)
.test_row_group_prune()
@@ -926,7 +930,7 @@ async fn prune_decimal_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(8)
.test_row_group_prune()
@@ -938,7 +942,7 @@ async fn prune_decimal_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(6)
.test_row_group_prune()
@@ -950,7 +954,7 @@ async fn prune_decimal_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(8)
.test_row_group_prune()
@@ -1026,7 +1030,7 @@ async fn prune_decimal_in_list() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(5)
.test_row_group_prune()
@@ -1038,7 +1042,7 @@ async fn prune_decimal_in_list() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(6)
.test_row_group_prune()
@@ -1050,7 +1054,7 @@ async fn prune_decimal_in_list() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(5)
.test_row_group_prune()
@@ -1062,7 +1066,7 @@ async fn prune_decimal_in_list() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(6)
.test_row_group_prune()
@@ -1177,7 +1181,9 @@ async fn prune_string_neq() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ // 'all frontends' is fully matched (value outside min/max) and skips Bloom.
+ // 'mixed' and 'all backends' are Bloom-evaluated; != cannot Bloom-prune.
+ .with_matched_by_bloom_filter(Some(2))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(14)
.test_row_group_prune()
@@ -1196,7 +1202,7 @@ async fn prune_string_lt() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(2))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(3)
.test_row_group_prune()
@@ -1211,7 +1217,7 @@ async fn prune_string_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
// all backends from 'mixed' and 'all backends'
.with_expected_rows(8)
@@ -1285,7 +1291,9 @@ async fn prune_binary_neq() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ // 'all frontends' is fully matched (value outside min/max) and skips Bloom.
+ // 'mixed' and 'all backends' are Bloom-evaluated; != cannot Bloom-prune.
+ .with_matched_by_bloom_filter(Some(2))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(14)
.test_row_group_prune()
@@ -1304,7 +1312,7 @@ async fn prune_binary_lt() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(2))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(3)
.test_row_group_prune()
@@ -1319,7 +1327,7 @@ async fn prune_binary_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
// all backends from 'mixed' and 'all backends'
.with_expected_rows(8)
@@ -1392,7 +1400,9 @@ async fn prune_fixedsizebinary_neq() {
.with_matched_by_stats(Some(3))
.with_pruned_by_stats(Some(0))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(3))
+ // 'all frontends' and 'all backends' are fully matched ('be1' outside min/max).
+ // Only 'mixed' contains 'be1' and is Bloom-evaluated.
+ .with_matched_by_bloom_filter(Some(1))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(14)
.test_row_group_prune()
@@ -1411,7 +1421,7 @@ async fn prune_fixedsizebinary_lt() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(2))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(2)
.test_row_group_prune()
@@ -1426,7 +1436,7 @@ async fn prune_fixedsizebinary_lt() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
// all backends from 'mixed' and 'all backends'
.with_expected_rows(8)
@@ -1447,7 +1457,8 @@ async fn prune_periods_in_column_names() {
.with_matched_by_stats(Some(2))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(2))
+ // All-frontend group is fully matched; only the mixed group is Bloom-evaluated.
+ .with_matched_by_bloom_filter(Some(1))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(7)
.test_row_group_prune()
@@ -1459,7 +1470,8 @@ async fn prune_periods_in_column_names() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(2))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ // Remaining group is all HTTP PUT, so statistics fully match != HTTP GET.
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(5)
.test_row_group_prune()
@@ -1494,7 +1506,7 @@ async fn test_row_group_with_null_values() {
.with_pruned_files(Some(0))
.with_pruned_by_stats(Some(2))
.with_expected_rows(5)
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.test_row_group_prune()
.await;
@@ -1508,7 +1520,7 @@ async fn test_row_group_with_null_values() {
.with_pruned_files(Some(0))
.with_pruned_by_stats(Some(1))
.with_expected_rows(10)
- .with_matched_by_bloom_filter(Some(2))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.test_row_group_prune()
.await;
@@ -1522,7 +1534,7 @@ async fn test_row_group_with_null_values() {
.with_pruned_files(Some(0))
.with_pruned_by_stats(Some(2))
.with_expected_rows(5)
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.test_row_group_prune()
.await;
@@ -2095,7 +2107,7 @@ async fn prune_like_prefix() {
.with_matched_by_stats(Some(1))
.with_pruned_by_stats(Some(1))
.with_pruned_files(Some(0))
- .with_matched_by_bloom_filter(Some(1))
+ .with_matched_by_bloom_filter(Some(0))
.with_pruned_by_bloom_filter(Some(0))
.with_expected_rows(1) // only "a" matches LIKE 'a%'
.test_row_group_prune()
diff --git a/datafusion/core/tests/sql/explain_analyze.rs b/datafusion/core/tests/sql/explain_analyze.rs
index 3efe084bae..4536a7f350 100644
--- a/datafusion/core/tests/sql/explain_analyze.rs
+++ b/datafusion/core/tests/sql/explain_analyze.rs
@@ -880,10 +880,7 @@ async fn parquet_explain_analyze() {
// should contain aggregated stats
assert_contains!(&formatted, "output_rows=8");
- assert_contains!(
- &formatted,
- "row_groups_pruned_bloom_filter=1 total \u{2192} 1 matched"
- );
+ assert_not_contains!(&formatted, "row_groups_pruned_bloom_filter");
assert_contains!(
&formatted,
"row_groups_pruned_statistics=1 total \u{2192} 1 matched"
@@ -896,16 +893,14 @@ async fn parquet_explain_analyze() {
// (file-> row-group -> page)
let i_file = formatted.find("files_ranges_pruned_statistics").unwrap();
let i_rowgroup_stat = formatted.find("row_groups_pruned_statistics").unwrap();
- let i_rowgroup_bloomfilter =
- formatted.find("row_groups_pruned_bloom_filter").unwrap();
let i_page_rows = formatted.find("page_index_rows_pruned").unwrap();
let i_page_pages = formatted.find("page_index_pages_pruned").unwrap();
assert!(
(i_file < i_rowgroup_stat)
- && (i_rowgroup_stat < i_rowgroup_bloomfilter)
- && (i_rowgroup_bloomfilter < i_page_pages && i_page_pages < i_page_rows),
- "The parquet pruning metrics should be displayed in an order of: file range -> row group statistics -> row group bloom filter -> page index."
+ && (i_rowgroup_stat < i_page_pages)
+ && (i_page_pages < i_page_rows),
+ "The parquet pruning metrics should be displayed in an order of: file range -> row group statistics -> page index."
);
}
@@ -1179,7 +1174,7 @@ async fn parquet_explain_analyze_verbose() {
.to_string();
// should contain the raw per file stats (with the label)
- assert_contains!(&formatted, "row_groups_pruned_bloom_filter{partition=0");
+ assert_not_contains!(&formatted, "row_groups_pruned_bloom_filter");
assert_contains!(&formatted, "row_groups_pruned_statistics{partition=0");
}
diff --git a/datafusion/datasource-parquet/src/opener/mod.rs b/datafusion/datasource-parquet/src/opener/mod.rs
index ddb5609c31..b048a23eea 100644
--- a/datafusion/datasource-parquet/src/opener/mod.rs
+++ b/datafusion/datasource-parquet/src/opener/mod.rs
@@ -1336,26 +1336,14 @@ impl FiltersPreparedParquetOpen {
.row_groups_pruned_statistics
.add_matched(row_groups.remaining_row_group_count());
}
-
- if !prepared.enable_bloom_filter || row_groups.is_empty() {
- // Update metrics: bloom filter unavailable, so all row groups are
- // matched (not pruned)
- prepared
- .file_metrics
- .row_groups_pruned_bloom_filter
- .add_matched(row_groups.remaining_row_group_count());
- }
} else {
// Update metrics: no predicate, so all row groups are matched (not pruned)
+ // by statistics. Bloom pruning did not run, so its metrics are unchanged.
let remaining = row_groups.remaining_row_group_count();
prepared
.file_metrics
.row_groups_pruned_statistics
.add_matched(remaining);
- prepared
- .file_metrics
- .row_groups_pruned_bloom_filter
- .add_matched(remaining);
}
Ok(RowGroupsPrunedParquetOpen {
@@ -3413,6 +3401,32 @@ mod test {
.expect("row_groups_pruned_statistics metric is emitted")
}
+ /// Bloom pruning counters after a scan, as `(pruned, matched)`.
+ ///
+ /// Direct `MetricsSet` lookup, not plan display: idle Bloom metrics are
+ /// omitted from displayed plans even when the counters remain registered.
+ ///
+ /// Opening one file still registers this name more than once:
+ /// `prepare_open_file` creates `ParquetFileMetrics`, and each
+ /// `ParquetFileReaderFactory::create_reader` call (initial reader plus
+ /// Bloom replacement reader) creates another independent set. Sum the
+ /// counters the same way `MetricsSet::sum_by_name` aggregates pruning
+ /// metrics. Only the opener's set is incremented by Bloom pruning; the
+ /// reader copies stay at zero.
+ fn bloom_filter_pruning_metrics(metrics: &ExecutionPlanMetricsSet) -> (usize, usize) {
+ use datafusion_physical_plan::metrics::MetricValue;
+ match metrics
+ .clone_inner()
+ .sum_by_name("row_groups_pruned_bloom_filter")
+ {
+ Some(MetricValue::PruningMetrics {
+ pruning_metrics, ..
+ }) => (pruning_metrics.pruned(), pruning_metrics.matched()),
+ Some(_) => panic!("row_groups_pruned_bloom_filter is not a pruning metric"),
+ None => panic!("row_groups_pruned_bloom_filter metric is registered"),
+ }
+ }
+
#[tokio::test]
async fn test_prune_all_null_column_equality_from_file_statistics() {
// Regression: a column whose file statistics say every value is
@@ -4743,6 +4757,16 @@ mod test {
// contribute to this metric.
bloom_bytes.push(counter_metric_value(&metrics, "bytes_scanned"));
assert_eq!(collect_int32_values(stream).await, vec![1, 1, 1, 1]);
+ let (pruned, matched) = bloom_filter_pruning_metrics(&metrics);
+ assert_eq!(pruned, 0);
+ if stats_pruning {
+ // [1,1,1] is fully matched by statistics and skips Bloom.
+ // [0,1,2] still evaluates Bloom for `a = 1` and cannot prune.
+ assert_eq!(matched, 1);
+ } else {
+ // Statistics pruning is off, so both row groups evaluate Bloom.
+ assert_eq!(matched, 2);
+ }
}
assert!(bloom_bytes[1] > 0, "partial row group needs Bloom I/O");
assert!(
@@ -4781,6 +4805,10 @@ mod test {
.unwrap();
assert_eq!(counter_metric_value(&metrics, "bytes_scanned"), 0);
assert_eq!(collect_int32_values(stream).await, vec![1, 1, 1]);
+ let (pruned, matched) = bloom_filter_pruning_metrics(&metrics);
+ // Bloom reads and evaluation are skipped for an entirely fully matched file.
+ assert_eq!(pruned, 0);
+ assert_eq!(matched, 0);
}
#[test]
diff --git a/datafusion/datasource-parquet/src/row_group_filter.rs b/datafusion/datasource-parquet/src/row_group_filter.rs
index fb7e78755d..b3feac2d3c 100644
--- a/datafusion/datasource-parquet/src/row_group_filter.rs
+++ b/datafusion/datasource-parquet/src/row_group_filter.rs
@@ -472,7 +472,8 @@ impl RowGroupAccessPlanFilter {
}
if self.access_plan.is_fully_matched(idx) {
- metrics.row_groups_pruned_bloom_filter.add_matched(1);
+ // Statistics already proved every row matches. Bloom did not
+ // evaluate this group, so do not record a Bloom outcome.
continue;
}
@@ -481,7 +482,6 @@ impl RowGroupAccessPlanFilter {
// evaluation in that case: it runs once per row group and can be expensive for wide
// predicates, a cost files written without bloom filters would pay for nothing.
if stats.is_empty() {
- metrics.row_groups_pruned_bloom_filter.add_matched(1);
continue;
}
@@ -493,7 +493,7 @@ impl RowGroupAccessPlanFilter {
"Error evaluating row group predicate on bloom filter: {e}"
);
metrics.predicate_evaluation_errors.add(1);
- false
+ continue;
}
};
@@ -622,9 +622,12 @@ mod tests {
use arrow::datatypes::DataType::Decimal128;
use arrow::datatypes::{DataType, Field};
- use datafusion_expr::{cast, col, lit};
- use datafusion_physical_expr::PhysicalExpr;
+ use datafusion_expr::{Operator, cast, col, lit};
use datafusion_physical_expr::planner::logical2physical;
+ use datafusion_physical_expr::{
+ PhysicalExpr,
+ expressions::{BinaryExpr, CastExpr, Column as PhysicalColumn, Literal},
+ };
use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
use datafusion_pruning::PruningPredicateBuilder;
use parquet::arrow::ArrowSchemaConverter;
@@ -1640,7 +1643,49 @@ mod tests {
// the row group with a bloom filter is still evaluated and pruned.
assert_pruned(row_groups, ExpectedPruning::Some(vec![0]));
assert_eq!(metrics.row_groups_pruned_bloom_filter.pruned(), 1);
- assert_eq!(metrics.row_groups_pruned_bloom_filter.matched(), 1);
+ assert_eq!(metrics.row_groups_pruned_bloom_filter.matched(), 0);
+ }
+
+ #[test]
+ fn bloom_filter_pruning_error_retains_row_group_without_match() {
+ let schema =
+ Arc::new(Schema::new(vec![Field::new("c1", DataType::Int32, false)]));
+ let cast_expr: Arc<dyn PhysicalExpr> = Arc::new(CastExpr::new(
+ Arc::new(PhysicalColumn::new("c1", 0)),
+ DataType::FixedSizeBinary(4),
+ None,
+ ));
+ let literal: Arc<dyn PhysicalExpr> = Arc::new(Literal::new(
+ ScalarValue::FixedSizeBinary(4, Some(vec![0, 0, 0, 1])),
+ ));
+ let expr = Arc::new(BinaryExpr::new(cast_expr, Operator::Eq, literal));
+ let pruning_predicate = PruningPredicateBuilder::new()
+ .with_file_schema(schema)
+ .try_build(expr)
+ .unwrap();
+
+ // A nonempty map bypasses the no-filter path. Evaluating the predicate
+ // then fails because Arrow cannot cast Int32 statistics to FixedSizeBinary.
+ let mut sbbf = Sbbf::new_with_ndv_fpp(10, 0.01).unwrap();
+ sbbf.insert(&1_i32);
+ let mut bloom_statistics = BloomFilterStatistics::new();
+ bloom_statistics.insert("loaded_filter", sbbf, PhysicalType::INT32, 4);
+ assert!(pruning_predicate.prune(&bloom_statistics).is_err());
+
+ let metrics = parquet_file_metrics();
+ let mut row_groups = RowGroupAccessPlanFilter::new(ParquetAccessPlan::new_all(1));
+ row_groups.prune_by_bloom_filters(
+ &pruning_predicate,
+ &metrics,
+ &[bloom_statistics],
+ );
+
+ // An evaluation failure is conservative: retain the row group, report
+ // the error separately, and do not call the result a Bloom match.
+ assert_pruned(row_groups, ExpectedPruning::Some(vec![0]));
+ assert_eq!(metrics.predicate_evaluation_errors.value(), 1);
+ assert_eq!(metrics.row_groups_pruned_bloom_filter.pruned(), 0);
+ assert_eq!(metrics.row_groups_pruned_bloom_filter.matched(), 0);
}
fn get_row_group_meta_data(
diff --git a/datafusion/physical-plan/src/display.rs b/datafusion/physical-plan/src/display.rs
index 37f21ad4d6..f13e66d7d0 100644
--- a/datafusion/physical-plan/src/display.rs
+++ b/datafusion/physical-plan/src/display.rs
@@ -21,6 +21,7 @@
use std::collections::{BTreeMap, HashMap};
use std::fmt;
use std::fmt::Formatter;
+use std::sync::Arc;
use std::time::Duration;
use serde::Serialize;
@@ -31,7 +32,7 @@ use datafusion_common::display::GraphvizBuilder;
use datafusion_expr::display_schema;
use datafusion_physical_expr::LexOrdering;
-use crate::metrics::{MetricCategory, MetricType, MetricValue};
+use crate::metrics::{MetricCategory, MetricType, MetricValue, MetricsSet};
use crate::render_tree::RenderTree;
use crate::operator_statistics::StatisticsRegistry;
@@ -537,6 +538,30 @@ enum ShowMetrics {
Full,
}
+const BLOOM_FILTER_PRUNING_METRIC_NAME: &str = "row_groups_pruned_bloom_filter";
+
+/// Omits an idle Bloom pruning metric from physical-plan display while
+/// preserving it in the underlying [`MetricsSet`].
+fn omit_idle_bloom_filter_pruning_metric(metrics: &MetricsSet) -> MetricsSet {
+ let mut filtered = MetricsSet::new();
+ for metric in metrics.iter() {
+ let is_idle_bloom_metric = matches!(
+ metric.value(),
+ MetricValue::PruningMetrics {
+ name,
+ pruning_metrics,
+ } if name == BLOOM_FILTER_PRUNING_METRIC_NAME
+ && pruning_metrics.pruned() == 0
+ && pruning_metrics.matched() == 0
+ && pruning_metrics.fully_matched() == 0
+ );
+ if !is_idle_bloom_metric {
+ filtered.push(Arc::clone(metric));
+ }
+ }
+ filtered
+}
+
/// Formats plans with a single line per node.
///
/// # Example
@@ -577,9 +602,10 @@ impl ExecutionPlanVisitor for IndentVisitor<'_, '_> {
ShowMetrics::None => {}
ShowMetrics::Aggregated => {
if let Some(metrics) = plan.metrics() {
- let mut metrics = metrics
+ let metrics = metrics
.filter_by_metric_types(self.metric_types)
- .aggregate_by_name()
+ .aggregate_by_name();
+ let mut metrics = omit_idle_bloom_filter_pruning_metric(&metrics)
.sorted_for_display()
.timestamps_removed();
if let Some(cats) = self.metric_categories {
@@ -595,7 +621,8 @@ impl ExecutionPlanVisitor for IndentVisitor<'_, '_> {
}
ShowMetrics::Full => {
if let Some(metrics) = plan.metrics() {
- let mut metrics = metrics.filter_by_metric_types(self.metric_types);
+ let metrics = metrics.filter_by_metric_types(self.metric_types);
+ let mut metrics = omit_idle_bloom_filter_pruning_metric(&metrics);
if let Some(cats) = self.metric_categories {
metrics = metrics.filter_by_categories(cats);
}
@@ -683,9 +710,10 @@ impl ExecutionPlanVisitor for GraphvizVisitor<'_, '_> {
ShowMetrics::None => "".to_string(),
ShowMetrics::Aggregated => {
if let Some(metrics) = plan.metrics() {
- let mut metrics = metrics
+ let metrics = metrics
.filter_by_metric_types(self.metric_types)
- .aggregate_by_name()
+ .aggregate_by_name();
+ let mut metrics = omit_idle_bloom_filter_pruning_metric(&metrics)
.sorted_for_display()
.timestamps_removed();
if let Some(cats) = self.metric_categories {
@@ -701,7 +729,8 @@ impl ExecutionPlanVisitor for GraphvizVisitor<'_, '_> {
}
ShowMetrics::Full => {
if let Some(metrics) = plan.metrics() {
- let mut metrics = metrics.filter_by_metric_types(self.metric_types);
+ let metrics = metrics.filter_by_metric_types(self.metric_types);
+ let mut metrics = omit_idle_bloom_filter_pruning_metric(&metrics);
if let Some(cats) = self.metric_categories {
metrics = metrics.filter_by_categories(cats);
}
@@ -895,12 +924,18 @@ impl PgJsonExecutionPlanVisitor<'_> {
let metrics = match self.show_metrics {
ShowMetrics::None => return,
- ShowMetrics::Aggregated => metrics
- .filter_by_metric_types(self.metric_types)
- .aggregate_by_name()
- .sorted_for_display()
- .timestamps_removed(),
- ShowMetrics::Full => metrics.filter_by_metric_types(self.metric_types),
+ ShowMetrics::Aggregated => {
+ let metrics = metrics
+ .filter_by_metric_types(self.metric_types)
+ .aggregate_by_name();
+ omit_idle_bloom_filter_pruning_metric(&metrics)
+ .sorted_for_display()
+ .timestamps_removed()
+ }
+ ShowMetrics::Full => {
+ let metrics = metrics.filter_by_metric_types(self.metric_types);
+ omit_idle_bloom_filter_pruning_metric(&metrics)
+ }
};
let metrics = if let Some(cats) = self.metric_categories {
metrics.filter_by_categories(cats)
@@ -1708,17 +1743,26 @@ mod tests {
}
mod pgjson {
+ use std::borrow::Cow;
use std::sync::Arc;
use std::time::Duration;
use arrow::datatypes::{DataType, Field, Schema};
use insta::assert_snapshot;
- use super::super::DisplayableExecutionPlan;
+ use super::super::{BLOOM_FILTER_PRUNING_METRIC_NAME, DisplayableExecutionPlan};
use crate::empty::EmptyExec;
use crate::filter::FilterExec;
+ use crate::metrics::{
+ Count, Metric, MetricValue, MetricsSet, PruningMetrics, Time,
+ };
use crate::projection::ProjectionExec;
- use crate::{ChildrenPropertiesMode, ExecutionPlan, ReplaceChildrenOptions};
+ use crate::{
+ ChildrenPropertiesMode, DisplayFormatType, ExecutionPlan, PlanProperties,
+ ReplaceChildrenOptions,
+ };
+ use datafusion_common::Result;
+ use datafusion_execution::{SendableRecordBatchStream, TaskContext};
use datafusion_physical_expr::expressions::{binary, col, lit};
use datafusion_physical_expr::{Partitioning, PhysicalExpr};
@@ -1742,6 +1786,142 @@ mod tests {
Arc::new(ProjectionExec::try_new(proj_expr, filter).unwrap())
}
+ /// Wraps a plan with a hand-crafted metric set so display behavior can
+ /// be tested without executing the plan.
+ #[derive(Debug)]
+ struct WithMetrics {
+ inner: Arc<dyn ExecutionPlan>,
+ metrics: MetricsSet,
+ }
+
+ impl crate::DisplayAs for WithMetrics {
+ fn fmt_as(
+ &self,
+ _t: DisplayFormatType,
+ f: &mut std::fmt::Formatter,
+ ) -> std::fmt::Result {
+ write!(f, "WithMetrics")
+ }
+ }
+
+ impl ExecutionPlan for WithMetrics {
+ fn name(&self) -> &'static str {
+ "WithMetrics"
+ }
+
+ fn properties(&self) -> &Arc<PlanProperties> {
+ self.inner.properties()
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
+ vec![&self.inner]
+ }
+
+ fn apply_expressions(
+ &self,
+ _f: &mut dyn FnMut(
+ &Arc<dyn PhysicalExpr>,
+ ) -> Result<
+ datafusion_common::tree_node::TreeNodeRecursion,
+ >,
+ ) -> Result<datafusion_common::tree_node::TreeNodeRecursion> {
+ Ok(datafusion_common::tree_node::TreeNodeRecursion::Continue)
+ }
+
+ fn replace_children(
+ self: Arc<Self>,
+ _: Vec<Arc<dyn ExecutionPlan>>,
+ _: ReplaceChildrenOptions,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ unimplemented!()
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ children: Vec<Arc<dyn ExecutionPlan>>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ self.replace_children(
+ children,
+ ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
+ )
+ }
+
+ fn execute(
+ &self,
+ _: usize,
+ _: Arc<TaskContext>,
+ ) -> Result<SendableRecordBatchStream> {
+ unimplemented!()
+ }
+
+ fn metrics(&self) -> Option<MetricsSet> {
+ Some(self.metrics.clone())
+ }
+ }
+
+ fn plan_with_metrics(metrics: MetricsSet) -> Arc<dyn ExecutionPlan> {
+ Arc::new(WithMetrics {
+ inner: sample_plan(),
+ metrics,
+ })
+ }
+
+ fn pruning_metric(
+ name: &'static str,
+ pruned: usize,
+ matched: usize,
+ ) -> Arc<Metric> {
+ pruning_metric_parts(name, pruned, matched, 0, None)
+ }
+
+ fn pruning_metric_parts(
+ name: &'static str,
+ pruned: usize,
+ matched: usize,
+ fully_matched: usize,
+ partition: Option<usize>,
+ ) -> Arc<Metric> {
+ let pruning_metrics = PruningMetrics::new();
+ pruning_metrics.add_pruned(pruned);
+ pruning_metrics.add_matched(matched);
+ pruning_metrics.add_fully_matched(fully_matched);
+ Arc::new(Metric::new(
+ MetricValue::PruningMetrics {
+ name: Cow::Borrowed(name),
+ pruning_metrics,
+ },
+ partition,
+ ))
+ }
+
+ fn indent_and_graphviz_metric_displays(plan: &dyn ExecutionPlan) -> [String; 4] {
+ [
+ DisplayableExecutionPlan::with_metrics(plan)
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(plan)
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_metrics(plan)
+ .graphviz()
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(plan)
+ .graphviz()
+ .to_string(),
+ ]
+ }
+
+ fn pgjson_extras(plan: &dyn ExecutionPlan, full: bool) -> serde_json::Value {
+ let display = if full {
+ DisplayableExecutionPlan::with_full_metrics(plan)
+ } else {
+ DisplayableExecutionPlan::with_metrics(plan)
+ };
+ let out = display.pgjson(false).to_string();
+ let value: serde_json::Value = serde_json::from_str(&out).unwrap();
+ value[0]["Plan"]["Extras"].clone()
+ }
+
#[test]
fn pgjson_renders_plan_without_metrics() {
let plan = sample_plan();
@@ -1767,78 +1947,6 @@ mod tests {
#[test]
fn pgjson_emits_pg_canonical_metric_keys() {
- use crate::metrics::{Count, Metric, MetricValue, MetricsSet, Time};
- use crate::{DisplayFormatType, ExecutionPlan, PlanProperties};
- use datafusion_common::Result;
- use datafusion_execution::{SendableRecordBatchStream, TaskContext};
-
- // Wrap `sample_plan()` with an adapter node that exposes a
- // hand-crafted `MetricsSet` so we can assert the PG key mapping
- // without running anything.
- #[derive(Debug)]
- struct WithMetrics {
- inner: Arc<dyn ExecutionPlan>,
- metrics: MetricsSet,
- }
- impl crate::DisplayAs for WithMetrics {
- fn fmt_as(
- &self,
- _t: DisplayFormatType,
- f: &mut std::fmt::Formatter,
- ) -> std::fmt::Result {
- write!(f, "WithMetrics")
- }
- }
- impl ExecutionPlan for WithMetrics {
- fn name(&self) -> &'static str {
- "WithMetrics"
- }
- fn properties(&self) -> &Arc<PlanProperties> {
- self.inner.properties()
- }
- fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
- vec![&self.inner]
- }
- fn apply_expressions(
- &self,
- _f: &mut dyn FnMut(
- &Arc<dyn PhysicalExpr>,
- ) -> Result<
- datafusion_common::tree_node::TreeNodeRecursion,
- >,
- ) -> Result<datafusion_common::tree_node::TreeNodeRecursion>
- {
- Ok(datafusion_common::tree_node::TreeNodeRecursion::Continue)
- }
-
- fn replace_children(
- self: Arc<Self>,
- _: Vec<Arc<dyn ExecutionPlan>>,
- _: ReplaceChildrenOptions,
- ) -> Result<Arc<dyn ExecutionPlan>> {
- unimplemented!()
- }
- fn with_new_children(
- self: Arc<Self>,
- children: Vec<Arc<dyn ExecutionPlan>>,
- ) -> Result<Arc<dyn ExecutionPlan>> {
- self.replace_children(
- children,
- ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
- )
- }
- fn execute(
- &self,
- _: usize,
- _: Arc<TaskContext>,
- ) -> Result<SendableRecordBatchStream> {
- unimplemented!()
- }
- fn metrics(&self) -> Option<MetricsSet> {
- Some(self.metrics.clone())
- }
- }
-
let mut metrics = MetricsSet::new();
let rows = Count::new();
rows.add(42);
@@ -1856,10 +1964,7 @@ mod tests {
None,
)));
- let plan: Arc<dyn ExecutionPlan> = Arc::new(WithMetrics {
- inner: sample_plan(),
- metrics,
- });
+ let plan = plan_with_metrics(metrics);
let out = DisplayableExecutionPlan::with_metrics(plan.as_ref())
.pgjson(false)
@@ -1905,6 +2010,221 @@ mod tests {
assert!(root.get("Extras").is_none());
}
+ #[test]
+ fn idle_bloom_pruning_metrics_are_omitted_from_plan_display() {
+ let mut idle_metrics = MetricsSet::new();
+ idle_metrics.push(pruning_metric(BLOOM_FILTER_PRUNING_METRIC_NAME, 0, 0));
+ idle_metrics.push(pruning_metric("page_index_pages_pruned", 0, 0));
+ idle_metrics.push(pruning_metric("limit_pruned_row_groups", 0, 0));
+ idle_metrics.push(Arc::new(Metric::new(
+ MetricValue::Count {
+ name: Cow::Borrowed("custom_metric"),
+ count: Count::new(),
+ },
+ None,
+ )));
+ let idle_plan = plan_with_metrics(idle_metrics);
+
+ for rendered in [
+ DisplayableExecutionPlan::with_metrics(idle_plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(idle_plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_metrics(idle_plan.as_ref())
+ .graphviz()
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(idle_plan.as_ref())
+ .graphviz()
+ .to_string(),
+ ] {
+ assert!(!rendered.contains(BLOOM_FILTER_PRUNING_METRIC_NAME));
+ assert!(rendered.contains("page_index_pages_pruned=0 total → 0 matched"));
+ assert!(rendered.contains("limit_pruned_row_groups=0 total → 0 matched"));
+ assert!(rendered.contains("custom_metric=0"));
+ }
+
+ for display in [
+ DisplayableExecutionPlan::with_metrics(idle_plan.as_ref()),
+ DisplayableExecutionPlan::with_full_metrics(idle_plan.as_ref()),
+ ] {
+ let out = display.pgjson(false).to_string();
+ let value: serde_json::Value = serde_json::from_str(&out).unwrap();
+ let extras = value[0]["Plan"]["Extras"].as_object().unwrap();
+ assert!(!extras.contains_key(BLOOM_FILTER_PRUNING_METRIC_NAME));
+ assert!(extras.contains_key("page_index_pages_pruned"));
+ assert!(extras.contains_key("limit_pruned_row_groups"));
+ assert_eq!(extras["custom_metric"].as_u64(), Some(0));
+ }
+
+ // Put the idle copy after the non-zero copy. Full PostgreSQL JSON
+ // keys metrics only by name, so this also verifies that the idle
+ // copy cannot overwrite the genuine result.
+ let mut mixed_metrics = MetricsSet::new();
+ mixed_metrics.push(pruning_metric(BLOOM_FILTER_PRUNING_METRIC_NAME, 0, 1));
+ mixed_metrics.push(pruning_metric(BLOOM_FILTER_PRUNING_METRIC_NAME, 0, 0));
+ let mixed_plan = plan_with_metrics(mixed_metrics);
+
+ for rendered in [
+ DisplayableExecutionPlan::with_metrics(mixed_plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(mixed_plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_metrics(mixed_plan.as_ref())
+ .graphviz()
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(mixed_plan.as_ref())
+ .graphviz()
+ .to_string(),
+ ] {
+ assert_eq!(
+ rendered.matches(BLOOM_FILTER_PRUNING_METRIC_NAME).count(),
+ 1
+ );
+ assert!(
+ rendered
+ .contains("row_groups_pruned_bloom_filter=1 total → 1 matched")
+ );
+ }
+
+ for display in [
+ DisplayableExecutionPlan::with_metrics(mixed_plan.as_ref()),
+ DisplayableExecutionPlan::with_full_metrics(mixed_plan.as_ref()),
+ ] {
+ let out = display.pgjson(false).to_string();
+ let value: serde_json::Value = serde_json::from_str(&out).unwrap();
+ assert_eq!(
+ value[0]["Plan"]["Extras"][BLOOM_FILTER_PRUNING_METRIC_NAME].as_str(),
+ Some("1 total → 1 matched")
+ );
+ }
+ }
+
+ #[test]
+ fn nonzero_bloom_pruned_or_fully_matched_metrics_are_visible() {
+ let mut pruned_metrics = MetricsSet::new();
+ pruned_metrics.push(pruning_metric(BLOOM_FILTER_PRUNING_METRIC_NAME, 3, 0));
+ let pruned_plan = plan_with_metrics(pruned_metrics);
+
+ for rendered in indent_and_graphviz_metric_displays(pruned_plan.as_ref()) {
+ assert!(
+ rendered
+ .contains("row_groups_pruned_bloom_filter=3 total → 0 matched")
+ );
+ }
+ for extras in [
+ pgjson_extras(pruned_plan.as_ref(), false),
+ pgjson_extras(pruned_plan.as_ref(), true),
+ ] {
+ assert_eq!(
+ extras[BLOOM_FILTER_PRUNING_METRIC_NAME].as_str(),
+ Some("3 total → 0 matched")
+ );
+ }
+
+ let mut fully_matched_metrics = MetricsSet::new();
+ fully_matched_metrics.push(pruning_metric_parts(
+ BLOOM_FILTER_PRUNING_METRIC_NAME,
+ 0,
+ 0,
+ 2,
+ None,
+ ));
+ let fully_matched_plan = plan_with_metrics(fully_matched_metrics);
+
+ for rendered in
+ indent_and_graphviz_metric_displays(fully_matched_plan.as_ref())
+ {
+ assert!(rendered.contains(
+ "row_groups_pruned_bloom_filter=0 total → 0 matched -> 2 fully matched"
+ ));
+ }
+ for extras in [
+ pgjson_extras(fully_matched_plan.as_ref(), false),
+ pgjson_extras(fully_matched_plan.as_ref(), true),
+ ] {
+ assert_eq!(
+ extras[BLOOM_FILTER_PRUNING_METRIC_NAME].as_str(),
+ Some("0 total → 0 matched -> 2 fully matched")
+ );
+ }
+ }
+
+ #[test]
+ fn idle_and_active_bloom_metrics_on_distinct_partitions() {
+ let mut metrics = MetricsSet::new();
+ metrics.push(pruning_metric_parts(
+ BLOOM_FILTER_PRUNING_METRIC_NAME,
+ 0,
+ 0,
+ 0,
+ Some(0),
+ ));
+ metrics.push(pruning_metric_parts(
+ BLOOM_FILTER_PRUNING_METRIC_NAME,
+ 0,
+ 2,
+ 0,
+ Some(1),
+ ));
+ let plan = plan_with_metrics(metrics);
+
+ let aggregated = [
+ DisplayableExecutionPlan::with_metrics(plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_metrics(plan.as_ref())
+ .graphviz()
+ .to_string(),
+ ];
+ for rendered in &aggregated {
+ assert_eq!(
+ rendered.matches(BLOOM_FILTER_PRUNING_METRIC_NAME).count(),
+ 1
+ );
+ assert!(
+ rendered
+ .contains("row_groups_pruned_bloom_filter=2 total → 2 matched")
+ );
+ assert!(!rendered.contains("partition=0"));
+ assert!(!rendered.contains("partition=1"));
+ }
+
+ let full = [
+ DisplayableExecutionPlan::with_full_metrics(plan.as_ref())
+ .indent(false)
+ .to_string(),
+ DisplayableExecutionPlan::with_full_metrics(plan.as_ref())
+ .graphviz()
+ .to_string(),
+ ];
+ for rendered in &full {
+ assert_eq!(
+ rendered.matches(BLOOM_FILTER_PRUNING_METRIC_NAME).count(),
+ 1
+ );
+ assert!(rendered.contains(
+ "row_groups_pruned_bloom_filter{partition=1}=2 total → 2 matched"
+ ));
+ assert!(!rendered.contains("partition=0"));
+ }
+
+ // PostgreSQL JSON extras are keyed by metric name, so partition
+ // labels are not retained; omit still keeps the active totals.
+ for extras in [
+ pgjson_extras(plan.as_ref(), false),
+ pgjson_extras(plan.as_ref(), true),
+ ] {
+ assert_eq!(
+ extras[BLOOM_FILTER_PRUNING_METRIC_NAME].as_str(),
+ Some("2 total → 2 matched")
+ );
+ }
+ }
+
#[test]
fn pgjson_includes_summary_when_set() {
let plan = sample_plan();
diff --git a/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt b/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
index 6a6bad99f0..0279e4120c 100644
--- a/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
+++ b/datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt
@@ -104,7 +104,7 @@ Plan with Metrics
03)----ProjectionExec: expr=[id@0 as id, value@1 as v, value@1 + id@0 as name], metrics=[output_rows=10, <slt:ignore>]
04)------FilterExec: value@1 > 3, metrics=[output_rows=10, <slt:ignore>, selectivity=100% (10/10)]
05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, metrics=[output_rows=10, <slt:ignore>]
-06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/test_data.parquet]]}, projection=[id, value], file_type=parquet, predicate=value@1 > 3 AND DynamicFilter [ value@1 IS NULL OR value@1 > 800 ], dynamic_rg_pruning=eligible, pruning_predicate=value_null_count@1 != row_count@2 AND value_max@0 > 3 AND (value_null_count@1 > 0 OR value_null_count@1 != row_count@2 AND value_max@0 > 800), required_guarante [...]
+06)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/test_data.parquet]]}, projection=[id, value], file_type=parquet, predicate=value@1 > 3 AND DynamicFilter [ value@1 IS NULL OR value@1 > 800 ], dynamic_rg_pruning=eligible, pruning_predicate=value_null_count@1 != row_count@2 AND value_max@0 > 3 AND (value_null_count@1 > 0 OR value_null_count@1 != row_count@2 AND value_max@0 > 800), required_guarante [...]
statement ok
set datafusion.explain.analyze_level = dev;
diff --git a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
index 10d25d7235..16c1b68b4f 100644
--- a/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
+++ b/datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
@@ -98,7 +98,7 @@ explain analyze select v from t order by v desc limit 3;
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[v@0 DESC], preserve_partitioning=[false], filter=[v@0 IS NULL OR v@0 > 12], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=DynamicFilter [ v@0 IS NULL OR v@0 > 12 ], sort_order_for_reorder=[v@0 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=v_null_count@0 > 0 OR v_null_count@0 != row_count@2 AND v_max@1 > 12, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, out [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=DynamicFilter [ v@0 IS NULL OR v@0 > 12 ], sort_order_for_reorder=[v@0 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=v_null_count@0 > 0 OR v_null_count@0 != row_count@2 AND v_max@1 > 12, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, out [...]
# `EXPLAIN ANALYZE` with a *static* predicate must surface the
# `row_filter_skipped_fully_matched` metric with a non-zero value — this
@@ -140,7 +140,7 @@ explain analyze select v from t where v >= 4 order by v;
----
Plan with Metrics
01)SortExec: expr=[v@0 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=11, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=v@0 >= 4, sort_order_for_reorder=[v@0 ASC NULLS LAST], pruning_predicate=v_null_count@1 != row_count@2 AND v_max@0 >= 4, required_guarantees=[], metrics=[output_rows=11, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_stat [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_row_group_pruning/data.parquet]]}, projection=[v], file_type=parquet, predicate=v@0 >= 4, sort_order_for_reorder=[v@0 ASC NULLS LAST], pruning_predicate=v_null_count@1 != row_count@2 AND v_max@0 >= 4, required_guarantees=[], metrics=[output_rows=11, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_stat [...]
statement ok
drop table t;
diff --git a/datafusion/sqllogictest/test_files/explain_analyze.slt b/datafusion/sqllogictest/test_files/explain_analyze.slt
index 511ba88ed3..b77a86e805 100644
--- a/datafusion/sqllogictest/test_files/explain_analyze.slt
+++ b/datafusion/sqllogictest/test_files/explain_analyze.slt
@@ -247,7 +247,7 @@ explain analyze select * from cat_tracking where species > 'M' AND s >= 50 order
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -262,7 +262,7 @@ explain analyze select * from cat_tracking where species > 'M' AND s >= 50 order
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -277,7 +277,7 @@ explain analyze select * from cat_tracking where species > 'M' AND s >= 50 order
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -559,7 +559,7 @@ EXPLAIN (ANALYZE, METRICS 'rows', LEVEL summary) select * from cat_tracking wher
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- Quoted-string METRICS with multiple categories ----
@@ -568,7 +568,7 @@ EXPLAIN (ANALYZE, METRICS 'rows,bytes', LEVEL summary) select * from cat_trackin
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- (METRICS 'timing', LEVEL summary) — timing metrics only ----
@@ -588,7 +588,7 @@ EXPLAIN (ANALYZE, METRICS 'rows,bytes', TIMING off, LEVEL summary) select * from
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- TIMING sugar: `METRICS 'rows', TIMING on` ↔ rows + timing ----
@@ -597,7 +597,7 @@ EXPLAIN (ANALYZE, METRICS 'rows', TIMING on, LEVEL summary) select * from cat_tr
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, elapsed_compute=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- SUMMARY sugar: `SUMMARY on` ↔ `LEVEL summary` ----
# Equivalent to METRICS 'rows', LEVEL summary above.
@@ -607,7 +607,7 @@ EXPLAIN (ANALYZE, METRICS 'rows', SUMMARY on) select * from cat_tracking where s
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- Statement option overrides session config ----
# Session says 'timing' but statement-level `METRICS 'rows'` wins.
@@ -620,7 +620,7 @@ EXPLAIN (ANALYZE, METRICS 'rows', LEVEL summary) select * from cat_tracking wher
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
# ---- pgjson format: structural golden with no metrics ----
@@ -682,7 +682,7 @@ EXPLAIN (ANALYZE, METRICS rows, LEVEL summary) select * from cat_tracking where
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/explain_analyze/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_c [...]
statement ok
reset datafusion.sql_parser.dialect;
diff --git a/datafusion/sqllogictest/test_files/limit_pruning.slt b/datafusion/sqllogictest/test_files/limit_pruning.slt
index dd04506ed3..5fc58ed75b 100644
--- a/datafusion/sqllogictest/test_files/limit_pruning.slt
+++ b/datafusion/sqllogictest/test_files/limit_pruning.slt
@@ -63,7 +63,7 @@ set datafusion.explain.analyze_level = summary;
query TT
explain analyze select * from tracking_data where species > 'M' AND s >= 50 limit 3;
----
-Plan with Metrics DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/limit_pruning/data.parquet]]}, projection=[species, s], limit=3, file_type=parquet, predicate=species@0 > M AND s@1 >= 50, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned [...]
+Plan with Metrics DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/limit_pruning/data.parquet]]}, projection=[species, s], limit=3, file_type=parquet, predicate=species@0 > M AND s@1 >= 50, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50, required_guarantees=[], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>, files_ranges_pruned [...]
statement ok
CREATE TABLE fully_matched_limit_source AS VALUES
@@ -120,7 +120,7 @@ explain analyze select * from tracking_data where species > 'M' AND s >= 50 orde
----
Plan with Metrics
01)SortExec: TopK(fetch=3), expr=[species@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[species@0 < Nlpine Sheep], metrics=[output_rows=3, elapsed_compute=<slt:ignore>, output_bytes=<slt:ignore>]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/limit_pruning/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_cou [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/limit_pruning/data.parquet]]}, projection=[species, s], file_type=parquet, predicate=species@0 > M AND s@1 >= 50 AND DynamicFilter [ species@0 < Nlpine Sheep ], sort_order_for_reorder=[species@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=species_null_count@1 != row_count@2 AND species_max@0 > M AND s_null_count@4 != row_count@2 AND s_max@3 >= 50 AND species_null_cou [...]
statement ok
drop table tracking_data;
diff --git a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
index 674f866fc6..3ad551f571 100644
--- a/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
+++ b/datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
@@ -206,7 +206,7 @@ EXPLAIN ANALYZE SELECT t FROM topk_pushdown ORDER BY t * t LIMIT 10;
----
Plan with Metrics
01)SortExec: TopK(fetch=10), expr=[t@0 * t@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[t@0 * t@0 < 1884329474306198481], metrics=[output_rows=10, output_batches=1, row_replacements=10]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_pushdown.parquet]]}, projection=[t], output_ordering=[t@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ t@0 * t@0 < 1884329474306198481 ], dynamic_rg_pruning=eligible, metrics=[output_rows=128, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=782 total → 782 matched, row_groups_pruned_bloom_f [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_pushdown.parquet]]}, projection=[t], output_ordering=[t@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ t@0 * t@0 < 1884329474306198481 ], dynamic_rg_pruning=eligible, metrics=[output_rows=128, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=782 total → 782 matched, page_index_pages_pruned=0 [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -268,7 +268,7 @@ EXPLAIN ANALYZE SELECT * FROM topk_single_col ORDER BY b DESC LIMIT 1;
----
Plan with Metrics
01)SortExec: TopK(fetch=1), expr=[b@1 DESC], preserve_partitioning=[false], filter=[b@1 IS NULL OR b@1 > bd], metrics=[output_rows=1, output_batches=1, row_replacements=1]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -319,7 +319,7 @@ EXPLAIN ANALYZE SELECT * FROM topk_multi_col ORDER BY b ASC NULLS LAST, a DESC L
----
Plan with Metrics
01)SortExec: TopK(fetch=2), expr=[b@1 ASC NULLS LAST, a@0 DESC], preserve_partitioning=[false], filter=[b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac)], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@ [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@ [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -388,8 +388,8 @@ FROM join_probe p INNER JOIN join_build AS build
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], projection=[a@3, b@4, c@2, e@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_ro [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_nul [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_ [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_nul [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -474,9 +474,9 @@ INNER JOIN nested_t3 ON nested_t2.c = nested_t3.d;
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(c@3, d@0)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
02)--HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, b@0)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t1.parquet]]}, projection=[a, x], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
-04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t2.parquet]]}, projection=[b, c, y], file_type=parquet, predicate=DynamicFilter [ b@0 >= aa AND b@0 <= ab AND b@0 IN (SET) ([aa, ab]) ], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 >= aa AND b_null_count@1 != row_count@2 AND b_min@3 <= ab AND (b_null_count@1 != row_count@2 AND b_min@3 <= aa AND aa <= b_max@0 [...]
-05)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t3.parquet]]}, projection=[d, z], file_type=parquet, predicate=DynamicFilter [ d@0 >= ca AND d@0 <= cb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= ca AND d_null_count@1 != row_count@2 AND d_min@3 <= cb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_s [...]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t1.parquet]]}, projection=[a, x], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t2.parquet]]}, projection=[b, c, y], file_type=parquet, predicate=DynamicFilter [ b@0 >= aa AND b@0 <= ab AND b@0 IN (SET) ([aa, ab]) ], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_max@0 >= aa AND b_null_count@1 != row_count@2 AND b_min@3 <= ab AND (b_null_count@1 != row_count@2 AND b_min@3 <= aa AND aa <= b_max@0 [...]
+05)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nested_t3.parquet]]}, projection=[d, z], file_type=parquet, predicate=DynamicFilter [ d@0 >= ca AND d@0 <= cb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= ca AND d_null_count@1 != row_count@2 AND d_min@3 <= cb, required_guarantees=[], metrics=[output_rows=2, output_batches=1, files_ranges_pruned_s [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -605,8 +605,8 @@ LIMIT 2;
Plan with Metrics
01)SortExec: TopK(fetch=2), expr=[e@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[e@0 < bb], metrics=[output_rows=2, output_batches=1, row_replacements=2]
02)--HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, d@0)], projection=[e@2], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_r [...]
-04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_probe.parquet]]}, projection=[d, e], file_type=parquet, predicate=DynamicFilter [ d@0 >= aa AND d@0 <= ab AND d@0 IN (SET) ([aa, ab]) ] AND DynamicFilter [ e@1 < bb ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= aa AND d_null_count@1 != row_count@2 AND d_min@3 <= ab AND (d_null_count@1 != row_count@2 AN [...]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file [...]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_join_probe.parquet]]}, projection=[d, e], file_type=parquet, predicate=DynamicFilter [ d@0 >= aa AND d@0 <= ab AND d@0 IN (SET) ([aa, ab]) ] AND DynamicFilter [ e@1 < bb ], dynamic_rg_pruning=eligible, pruning_predicate=d_null_count@1 != row_count@2 AND d_max@0 >= aa AND d_null_count@1 != row_count@2 AND d_min@3 <= ab AND (d_null_count@1 != row_count@2 AN [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -656,7 +656,7 @@ EXPLAIN ANALYZE SELECT b, a FROM topk_proj ORDER BY a LIMIT 2;
Plan with Metrics
01)ProjectionExec: expr=[b@1 as b, a@0 as a], metrics=[output_rows=2, output_batches=1]
02)--SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 2], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_grou [...]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_grou [...]
# Case 2: prune — `SELECT a` — filter stays as `a < 2` on the scan.
query TT
@@ -664,7 +664,7 @@ EXPLAIN ANALYZE SELECT a FROM topk_proj ORDER BY a LIMIT 2;
----
Plan with Metrics
01)SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 2], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pr [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[a], file_type=parquet, predicate=DynamicFilter [ a@0 < 2 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_min@0 < 2, required_guarantees=[], metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pr [...]
# Case 3: expression — `SELECT a+1 AS a_plus_1` — the TopK filter is on
# `a_plus_1`, the scan predicate must read `a@0 + 1`.
@@ -673,7 +673,7 @@ EXPLAIN ANALYZE SELECT a + 1 AS a_plus_1, b FROM topk_proj ORDER BY a_plus_1 LIM
----
Plan with Metrics
01)SortExec: TopK(fetch=2), expr=[a_plus_1@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a_plus_1@0 < 3], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a_plus_1, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a_plus_1, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matche [...]
# Case 4: alias shadowing — `SELECT a+1 AS a` — the projection renames
# `a+1` to `a`, so the TopK's `a < 3` must still be rewritten to
@@ -683,7 +683,7 @@ EXPLAIN ANALYZE SELECT a + 1 AS a, b FROM topk_proj ORDER BY a LIMIT 2;
----
Plan with Metrics
01)SortExec: TopK(fetch=2), expr=[a@0 ASC NULLS LAST], preserve_partitioning=[false], filter=[a@0 < 3], metrics=[output_rows=2, output_batches=1, row_replacements=2]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_ [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_proj.parquet]]}, projection=[CAST(a@0 AS Int64) + 1 as a, b], file_type=parquet, predicate=DynamicFilter [ CAST(a@0 AS Int64) + 1 < 3 ], sort_order_for_reorder=[a@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -740,12 +740,12 @@ INNER JOIN (
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0)], projection=[a@0, min_value@2], metrics=[output_rows=2, output_batches=2, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_build.parquet]]}, projection=[a], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
03)--ProjectionExec: expr=[a@0 as a, min(join_agg_probe.value)@1 as min_value], metrics=[output_rows=2, output_batches=2]
04)----AggregateExec: mode=FinalPartitioned, gby=[a@0 as a], aggr=[min(join_agg_probe.value)], metrics=[output_rows=2, output_batches=2, spill_count=0, spilled_rows=0]
05)------RepartitionExec: partitioning=Hash([a@0], 4), input_partitions=1, metrics=[output_rows=2, output_batches=2, spill_count=0, spilled_rows=0]
06)--------AggregateExec: mode=Partial, gby=[a@0 as a], aggr=[min(join_agg_probe.value)], metrics=[output_rows=2, output_batches=1, spill_count=0, spilled_rows=0, skipped_aggregation_rows=0, reduction_factor=100% (2/2)]
-07)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_probe.parquet]]}, projection=[a, value], file_type=parquet, predicate=DynamicFilter [ a@0 >= h1 AND a@0 <= h2 AND a@0 IN (SET) ([h1, h2]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= h1 AND a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND (a_null_count@1 != row_count@2 AND a_min@3 <= h1 AND h1 [...]
+07)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/join_agg_probe.parquet]]}, projection=[a, value], file_type=parquet, predicate=DynamicFilter [ a@0 >= h1 AND a@0 <= h2 AND a@0 IN (SET) ([h1, h2]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= h1 AND a_null_count@1 != row_count@2 AND a_min@3 <= h2 AND (a_null_count@1 != row_count@2 AND a_min@3 <= h1 AND h1 [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -807,8 +807,8 @@ ON nulls_build.a = nulls_probe.a AND nulls_build.b = nulls_probe.b;
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=1, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=3, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_probe.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= 1 AND b@1 <= 2 AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:1}, {c0:,c1:2}, {c0:ab,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab A [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=3, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nulls_probe.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= 1 AND b@1 <= 2 AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:1}, {c0:,c1:2}, {c0:ab,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab A [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -873,8 +873,8 @@ ON lj_build.a = lj_probe.a AND lj_build.b = lj_probe.b;
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Left, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_ [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_ [...]
# LEFT SEMI JOIN: only matching build rows are returned; probe scan still
# receives the dynamic filter.
@@ -889,8 +889,8 @@ WHERE EXISTS (
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=LeftSemi, on=[(a@0, a@0), (b@1, b@1)], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=2, input_rows=4, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_cou [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/lj_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}, {c0:ab,c1:bb}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_cou [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -959,8 +959,8 @@ FROM hl_probe p INNER JOIN hl_build AS build
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], projection=[a@3, b@4, c@2, e@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_ [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_ [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_build.parquet]]}, projection=[a, b, c], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_op [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/hl_probe.parquet]]}, projection=[a, b, e], file_type=parquet, predicate=DynamicFilter [ a@0 >= aa AND a@0 <= ab AND b@1 >= ba AND b@1 <= bb AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@1 != row_count@2 AND a_max@0 >= aa AND a_null_count@1 != row_count@2 AND a_min@3 <= ab AND b_null_count@5 != row_count@2 AND b_max@4 >= ba AND b_ [...]
statement ok
drop table hl_build;
@@ -1008,8 +1008,8 @@ FROM int_build b INNER JOIN int_probe p
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id1@0, id1@0), (id2@1, id2@1)], projection=[id1@0, id2@1, value@2, data@5], metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_build.parquet]]}, projection=[id1, id2, value], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pr [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_probe.parquet]]}, projection=[id1, id2, data], file_type=parquet, predicate=DynamicFilter [ id1@0 >= 1 AND id1@0 <= 2 AND id2@1 >= 10 AND id2@1 <= 20 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=id1_null_count@1 != row_count@2 AND id1_max@0 >= 1 AND id1_null_count@1 != row_count@2 AND id1_min@3 <= 2 AND id2_null_count@5 != row_count@2 AN [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_build.parquet]]}, projection=[id1, id2, value], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0 [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/int_probe.parquet]]}, projection=[id1, id2, data], file_type=parquet, predicate=DynamicFilter [ id1@0 >= 1 AND id1@0 <= 2 AND id2@1 >= 10 AND id2@1 <= 20 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=id1_null_count@1 != row_count@2 AND id1_max@0 >= 1 AND id1_null_count@1 != row_count@2 AND id1_min@3 <= 2 AND id2_null_count@5 != row_count@2 AN [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -1060,8 +1060,8 @@ EXPLAIN ANALYZE SELECT nej_build.id, nej_probe.id FROM nej_build JOIN nej_probe
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=2, avg_fanout=100% (2/2), probe_hit_rate=100% (2/2)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_grou [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR id_null_count@0 != row_count@2 AND id_max@1 >= 11 AND id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@0 != row_co [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_e [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nej_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 IS NULL OR id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@0 > 0 OR id_null_count@0 != row_count@2 AND id_max@1 >= 11 AND id_null_count@0 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@0 != row_co [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -1103,8 +1103,8 @@ EXPLAIN ANALYZE SELECT mnej_build.a, mnej_build.b, mnej_probe.a, mnej_probe.b FR
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, a@0), (b@1, b@1)], NullsEqual: true, metrics=[output_rows=2, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=3, avg_fanout=100% (2/2), probe_hit_rate=66.67% (2/3)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_g [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 IS NULL OR b@1 IS NULL OR a@0 >= 1 AND a@0 <= 2 AND b@1 >= 10 AND b@1 <= 10 AND struct(a@0, b@1) IN (SET) ([{c0:1,c1:10}, {c0:2,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@0 > 0 OR b_null_count@1 > 0 OR a_null_count@0 != row_count@3 AND a_max@2 >= [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_build.parquet]]}, projection=[a, b], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_ope [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/mnej_probe.parquet]]}, projection=[a, b], file_type=parquet, predicate=DynamicFilter [ a@0 IS NULL OR b@1 IS NULL OR a@0 >= 1 AND a@0 <= 2 AND b@1 >= 10 AND b@1 <= 10 AND struct(a@0, b@1) IN (SET) ([{c0:1,c1:10}, {c0:2,c1:}]) ], dynamic_rg_pruning=eligible, pruning_predicate=a_null_count@0 > 0 OR b_null_count@1 > 0 OR a_null_count@0 != row_count@3 AND a_max@2 >= [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -1144,8 +1144,8 @@ EXPLAIN ANALYZE SELECT nnb_build.id, nnb_probe.id FROM nnb_build JOIN nnb_probe
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=1, output_batches=1, array_map_created_count=1, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_grou [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 22 AND id@0 IN (SET) ([11, 22]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_e [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnb_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 22 AND id@0 IN (SET) ([11, 22]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 22 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_max [...]
statement ok
reset datafusion.explain.analyze_categories;
@@ -1185,8 +1185,8 @@ EXPLAIN ANALYZE SELECT nnp_build.id, nnp_probe.id FROM nnp_build JOIN nnp_probe
----
Plan with Metrics
01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(id@0, id@0)], NullsEqual: true, metrics=[output_rows=1, output_batches=1, array_map_created_count=0, build_input_batches=1, build_input_rows=2, input_batches=1, input_rows=1, avg_fanout=100% (1/1), probe_hit_rate=100% (1/1)]
-02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_grou [...]
-03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_m [...]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_build.parquet]]}, projection=[id], file_type=parquet, metrics=[output_rows=2, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_e [...]
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/nnp_probe.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ id@0 >= 11 AND id@0 <= 11 AND id@0 IN (SET) ([11, NULL]) ], dynamic_rg_pruning=eligible, pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 11 AND id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND (id_null_count@1 != row_count@2 AND id_min@3 <= 11 AND 11 <= id_m [...]
statement ok
reset datafusion.explain.analyze_categories;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org