You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/15 21:43:28 UTC

(datafusion) 01/01: fix: suppress nondeterministic metrics in agg_dyn_e2e sqllogictest (#21657)

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch gh-readonly-queue/main/pr-21657-7bf39b5e747eb331a0da636198cfd9aaab15140f
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit 240fbdb915a25aaea1ffe0288909d8d54bbfd9f6
Author: Matt Butrovich <mb...@users.noreply.github.com>
AuthorDate: Wed Apr 15 17:43:05 2026 -0400

    fix: suppress nondeterministic metrics in agg_dyn_e2e sqllogictest (#21657)
    
    ## Which issue does this PR close?
    
    - Closes #.
    
    ## Rationale for this change
    
    PR #21620 (commit 5c653bee5) ported
    `test_aggregate_dynamic_filter_parquet_e2e` from Rust to sqllogictest
    using `analyze_categories = 'rows'`, which includes exact pushdown
    metrics. These metrics are nondeterministic under parallel execution —
    the order in which Partial aggregates publish dynamic filter updates
    races against when the scan reads each partition — so the expected
    output is flaky.
    
    Noticed on #21629 ([CI
    run](https://github.com/apache/datafusion/actions/runs/24472961090/job/71518239089?pr=21629))
    and confirmed on main ([CI
    run](https://github.com/apache/datafusion/actions/runs/24467213913/job/71497866154)).
    
    ## What changes are included in this PR?
    
    Switch the `agg_dyn_e2e` test to `analyze_level = summary` +
    `analyze_categories = 'none'`, suppressing nondeterministic metrics.
    This matches the approach already used by the other aggregate dynamic
    filter tests in the same file. The original Rust test only asserted
    `matched < 4` (i.e. some pruning happened); the important invariant —
    that the `DynamicFilter [ column1@0 > 4 ]` text and pruning predicate
    are correct — is still verified.
    
    ## Are these changes tested?
    
    Yes — the test itself is what is being fixed.
    
    ## Are there any user-facing changes?
    
    No.
---
 .../test_files/push_down_filter_regression.slt      | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/datafusion/sqllogictest/test_files/push_down_filter_regression.slt b/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
index 51ad998c02..7334054ff2 100644
--- a/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
+++ b/datafusion/sqllogictest/test_files/push_down_filter_regression.slt
@@ -218,21 +218,32 @@ LOCATION 'test_files/scratch/push_down_filter_regression/agg_dyn/';
 statement ok
 set datafusion.execution.collect_statistics = true;
 
+# Suppress metrics: pruning counts are nondeterministic under parallel
+# execution (the order in which Partial aggregates publish dynamic filter
+# updates races against when the scan reads each partition). The original
+# Rust test only asserted matched < 4; the important invariant here is
+# that the DynamicFilter text is correct.
 statement ok
-set datafusion.explain.analyze_categories = 'rows';
+set datafusion.explain.analyze_level = summary;
+
+statement ok
+set datafusion.explain.analyze_categories = 'none';
 
 query TT
 EXPLAIN ANALYZE select max(column1) from agg_dyn_e2e where column1 > 1;
 ----
 Plan with Metrics
-01)AggregateExec: mode=Final, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[output_rows=1, output_batches=1]
-02)--CoalescePartitionsExec, metrics=[output_rows=2, output_batches=2]
-03)----AggregateExec: mode=Partial, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[output_rows=2, output_batches=2]
-04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_0.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_2.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_3.parquet]] [...]
+01)AggregateExec: mode=Final, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[]
+02)--CoalescePartitionsExec, metrics=[]
+03)----AggregateExec: mode=Partial, gby=[], aggr=[max(agg_dyn_e2e.column1)], metrics=[]
+04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_0.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_2.parquet, WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_regression/agg_dyn/file_3.parquet]] [...]
 
 statement ok
 reset datafusion.explain.analyze_categories;
 
+statement ok
+reset datafusion.explain.analyze_level;
+
 statement ok
 reset datafusion.execution.collect_statistics;
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org