You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/15 21:43:28 UTC
(datafusion) 01/01: fix: suppress nondeterministic metrics in agg_dyn_e2e sqllogictest (#21657)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch gh-readonly-queue/main/pr-21657-7bf39b5e747eb331a0da636198cfd9aaab15140f
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 240fbdb915a25aaea1ffe0288909d8d54bbfd9f6
Author: Matt Butrovich <mb...@users.noreply.github.com>
AuthorDate: Wed Apr 15 17:43:05 2026 -0400
fix: suppress nondeterministic metrics in agg_dyn_e2e sqllogictest (#21657)
## Which issue does this PR close?
- Closes #.
## Rationale for this change
PR #21620 (commit 5c653bee5) ported
`test_aggregate_dynamic_filter_parquet_e2e` from Rust to sqllogictest
using `analyze_categories = 'rows'`, which includes exact pushdown
metrics. These metrics are nondeterministic under parallel execution —
the order in which Partial aggregates publish dynamic filter updates
races against when the scan reads each partition — so the expected
output is flaky.
Noticed on #21629 ([CI
run](https://github.com/apache/datafusion/actions/runs/24472961090/job/71518239089?pr=21629))
and confirmed on main ([CI
run](https://github.com/apache/datafusion/actions/runs/24467213913/job/71497866154)).
## What changes are included in this PR?
Switch the `agg_dyn_e2e` test to `analyze_level = summary` +
`analyze_categories = 'none'`, suppressing nondeterministic metrics.
This matches the approach already used by the other aggregate dynamic
filter tests in the same file. The original Rust test only asserted
`matched < 4` (i.e. some pruning happened); the important invariant —
that the `DynamicFilter [ column1@0 > 4 ]` text and pruning predicate
are correct — is still verified.
## Are these changes tested?
Yes — the test itself is what is being fixed.
## Are there any user-facing changes?
No.
---
.../test_files/push_down_filter_regression.slt | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/datafusion/sqllogictest/test_files/push_down_filter_regression.slt b/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
index 51ad998c02..7334054ff2 100644
--- a/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
+++ b/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
@@ -218,21 +218,32 @@ LOCATION 'test_files/scratch/push_down_filter_regression/agg_dyn/';
statement ok
set datafusion.execution.collect_statistics = true;
+# Suppress metrics: pruning counts are nondeterministic under parallel
+# execution (the order in which Partial aggregates publish dynamic filter
+# updates races against when the scan reads each partition). The original
+# Rust test only asserted matched < 4; the important invariant here is
+# that the DynamicFilter text is correct.
statement ok
-set datafusion.explain.analyze_categories = 'rows';
+set datafusion.explain.analyze_level = summary;
+
+statement ok
+set datafusion.explain.analyze_categories = 'none';
query TT
EXPLAIN ANALYZE select max(column1) from agg_dyn_e2e where column1 > 1;
----
Plan with Metrics
-01)AggregateExec: mode=Final, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[output_rows=1, output_batches=1]
-02)--CoalescePartitionsExec, metrics=[output_rows=2, output_batches=2]
-03)----AggregateExec: mode=Partial, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[output_rows=2, output_batches=2]
-04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_0.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_2.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_3.parquet]] [...]
+01)AggregateExec: mode=Final, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[]
+02)--CoalescePartitionsExec, metrics=[]
+03)----AggregateExec: mode=Partial, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[]
+04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_0.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_2.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_3.parquet]] [...]
statement ok
reset datafusion.explain.analyze_categories;
+statement ok
+reset datafusion.explain.analyze_level;
+
statement ok
reset datafusion.execution.collect_statistics;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org