You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/10/18 02:53:11 UTC

[PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

jayzhan211 opened a new pull request, #26461:
URL: https://github.com/apache/datafusion/pull/26461

   ## Which issue does this PR close?
   
   - Closes #26460.
   
   Related: #26171 (option 3), #26351, #26360, #25688.
   
   ## Rationale for this change
   
   `SanityCheckPlan` is the last rule in `PhysicalOptimizer::new()`, so rules appended with `SessionStateBuilder::with_physical_optimizer_rule` run after it and are never checked. On `main`, an appended rule that removes a window's sort silently returns wrong `row_number()` values. Custom pipelines have to keep the rule last by hand, and a custom list without it runs plans whose requirements do not hold. When the check does fail, the error only says `SanityCheckPlan`, so finding the rule that broke the plan has needed a manual bisect each time (for example #26391, #17077, #25921).
   
   #26171 agreed (option 3) that a custom pipeline must keep the plan valid, while optimization coverage is not guaranteed. This PR makes that validity part checked for every pipeline. The review on #26360 asked for the same shape: one rule, `EnsureRequirements`, makes the plan valid, and every later rule keeps it valid.
   
   ## What changes are included in this PR?
   
   - `SanityCheckPlan` leaves the default rule list. `DefaultPhysicalPlanner::optimize_physical_plan` runs the same check (unbounded inputs, ordering, distribution, `check_invariants(Executable)`) after the last rule, including appended rules. It replaces the planner's own `InvariantChecker(Executable)` walk, which `SanityCheckPlan` repeated.
   - The planner keeps the plan after each rule (one `Arc` per rule). When the ordering or distribution requirements fail, the error names the rule that broke them:
   
     ```text
     SanityCheckPlan: order and distribution requirements stopped holding after physical optimizer rule 'ProjectionPushdown'
     caused by
     Error during planning: Plan: [...] does not satisfy order requirements: [...]
     ```
   
     or, when the plan was already invalid as `EnsureRequirements` produced it:
   
     ```text
     SanityCheckPlan: the plan produced by physical optimizer rule 'EnsureRequirements' does not satisfy the order and distribution requirements
     ```
   
     Plans before `EnsureRequirements` need not satisfy the requirements, so only plans from its last run onwards are searched. The rule is recognized by its name; a separate enforcement phase (#25688) would give a cleaner boundary. Errors about unbounded inputs keep the plain `SanityCheckPlan` context, so existing messages do not change.
   - `sanity_checker` exposes `check_plan_tree_sanity`, `check_input_requirements` and `input_requirements_hold`, which the planner uses.
   - The upgrade guide, the optimizer rule reference and `explain.slt` are updated.
   
   ## What is the testing strategy for this PR?
   
   - `sanity_checker.rs`: a rule appended after the default rules that removes sorts is reported by name, both with the default list and with a custom list; a list without `EnsureRequirements` reports that no rule produced a valid plan; the default list passes.
   - `memory_limit/mod.rs`: the two sort merge join tests ran a `SortMergeJoinExec` over unsorted streams through a custom rule list without `EnsureRequirements`. They now sort the generated data by the join keys and declare that order, so the plan is valid and the tests still measure the join's memory use.
   - The extended workspace suite passes.
   
   ## Are there any user-facing changes?
   
   - Plans produced by rules added with `with_physical_optimizer_rule`, or by a custom rule list, are now checked. A plan whose requirements do not hold, such as a sort merge join over unsorted input, fails to plan instead of running and possibly returning wrong results.
   - `PhysicalOptimizer::new()` no longer contains `SanityCheckPlan`, and `EXPLAIN VERBOSE` no longer prints `physical_plan after SanityCheckPlan`. Code that looks the rule up in the default list no longer finds it. `SanityCheckPlan` itself remains available.
   - Errors for broken ordering or distribution requirements name the rule that broke them.
   
   This changes the public behavior of the default rule list, so it needs the `api change` label.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6251165213

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6250437807)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing planner-final-plan-check (2689cbbfae6c63ce9073b0a95cf09c4328fc75d7) to 290ad15 (merge-base) [diff](https://github.com/apache/datafusion/compare/290ad15cc842167ca4d3c0774ccdf3a1d9d61862..2689cbbfae6c63ce9073b0a95cf09c4328fc75d7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   group                                                 HEAD                                    planner-final-plan-check
   -----                                                 ----                                    ------------------------
   logical_aggregate_with_join                           1.00    345.2±1.80µs        ? ?/sec     1.00    344.1±2.25µs        ? ?/sec
   logical_correlated_subquery_exists                    1.01    206.3±0.75µs        ? ?/sec     1.00    205.0±1.41µs        ? ?/sec
   logical_correlated_subquery_in                        1.02    210.1±1.50µs        ? ?/sec     1.00    205.8±1.66µs        ? ?/sec
   logical_distinct_many_columns                         1.01    291.8±1.48µs        ? ?/sec     1.00    287.8±1.19µs        ? ?/sec
   logical_join_4_with_agg_and_filter                    1.00    193.3±1.01µs        ? ?/sec     1.00    192.9±2.53µs        ? ?/sec
   logical_join_8_with_agg_sort_limit                    1.01    341.5±1.48µs        ? ?/sec     1.00    337.7±1.90µs        ? ?/sec
   logical_join_chain_16                                 1.00    615.7±2.88µs        ? ?/sec     1.00    612.7±1.68µs        ? ?/sec
   logical_join_chain_4                                  1.03     75.3±0.34µs        ? ?/sec     1.00     73.1±0.25µs        ? ?/sec
   logical_join_chain_8                                  1.01    199.5±0.56µs        ? ?/sec     1.00    198.3±0.66µs        ? ?/sec
   logical_multiple_subqueries                           1.02    394.7±1.95µs        ? ?/sec     1.00    387.7±1.35µs        ? ?/sec
   logical_nested_cte_4_levels                           1.01    196.6±1.59µs        ? ?/sec     1.00    194.9±1.07µs        ? ?/sec
   logical_plan_struct_join_agg_sort                     1.05    125.2±0.52µs        ? ?/sec     1.00    119.7±0.93µs        ? ?/sec
   logical_plan_tpcds_all                                1.01     73.9±1.09ms        ? ?/sec     1.00     73.4±0.44ms        ? ?/sec
   logical_plan_tpch_all                                 1.02      5.3±0.05ms        ? ?/sec     1.00      5.2±0.03ms        ? ?/sec
   logical_scalar_subquery                               1.01    217.1±1.16µs        ? ?/sec     1.00    213.9±0.74µs        ? ?/sec
   logical_select_all_from_1000                          1.00      6.8±0.02ms        ? ?/sec     1.01      6.9±0.09ms        ? ?/sec
   logical_select_one_from_700                           1.00    228.5±1.46µs        ? ?/sec     1.01    230.9±1.94µs        ? ?/sec
   logical_trivial_join_high_numbered_columns            1.00    205.8±1.28µs        ? ?/sec     1.01    208.5±1.23µs        ? ?/sec
   logical_trivial_join_low_numbered_columns             1.00    194.9±1.28µs        ? ?/sec     1.01    196.8±1.13µs        ? ?/sec
   logical_union_4_branches                              1.01    308.6±2.04µs        ? ?/sec     1.00    306.3±1.69µs        ? ?/sec
   logical_union_8_branches                              1.00    620.9±2.79µs        ? ?/sec     1.01    627.2±6.31µs        ? ?/sec
   logical_wide_aggregate_1000_exprs                     1.01     42.6±0.27ms        ? ?/sec     1.00     42.1±0.22ms        ? ?/sec
   logical_wide_aggregate_100_exprs                      1.00  1559.8±10.77µs        ? ?/sec     1.00   1557.0±9.29µs        ? ?/sec
   logical_wide_case_50_exprs                            1.00  1718.9±14.38µs        ? ?/sec     1.00   1713.6±4.99µs        ? ?/sec
   logical_wide_filter_200_predicates                    1.00  1308.1±12.84µs        ? ?/sec     1.00   1301.7±5.19µs        ? ?/sec
   logical_wide_filter_50_predicates                     1.01    341.8±1.79µs        ? ?/sec     1.00    339.3±1.16µs        ? ?/sec
   optimizer_correlated_exists                           1.00    217.4±1.25µs        ? ?/sec     1.01    218.7±1.17µs        ? ?/sec
   optimizer_join_4_with_agg_filter                      1.00    393.9±1.91µs        ? ?/sec     1.03    403.8±1.80µs        ? ?/sec
   optimizer_join_chain_4                                1.00    154.0±0.31µs        ? ?/sec     1.01    156.2±0.31µs        ? ?/sec
   optimizer_join_chain_8                                1.01    537.3±2.32µs        ? ?/sec     1.00    533.9±0.79µs        ? ?/sec
   optimizer_select_all_from_1000                        1.00      6.8±0.02ms        ? ?/sec     1.00      6.8±0.02ms        ? ?/sec
   optimizer_select_one_from_700                         1.00    239.8±2.79µs        ? ?/sec     1.01    243.3±0.74µs        ? ?/sec
   optimizer_tpcds_all                                   1.01    254.5±1.85ms        ? ?/sec     1.00    252.7±1.01ms        ? ?/sec
   optimizer_tpch_all                                    1.00     15.4±0.08ms        ? ?/sec     1.00     15.4±0.09ms        ? ?/sec
   optimizer_wide_aggregate_100                          1.00   1751.6±4.11µs        ? ?/sec     1.00   1754.1±3.23µs        ? ?/sec
   optimizer_wide_filter_200                             1.02      4.1±0.04ms        ? ?/sec     1.00      4.0±0.02ms        ? ?/sec
   physical_intersection                                 1.00    510.1±2.04µs        ? ?/sec     1.00    509.2±2.72µs        ? ?/sec
   physical_join_consider_sort                           1.00    951.2±2.27µs        ? ?/sec     1.03   981.9±10.35µs        ? ?/sec
   physical_join_distinct                                1.00    187.1±1.20µs        ? ?/sec     1.02    190.3±1.33µs        ? ?/sec
   physical_many_self_joins                              1.00      7.4±0.04ms        ? ?/sec     1.02      7.5±0.04ms        ? ?/sec
   physical_plan_clickbench_all                          1.06    154.2±9.17ms        ? ?/sec     1.00    145.5±4.13ms        ? ?/sec
   physical_plan_clickbench_q1                           1.00  1380.1±123.75µs        ? ?/sec    1.05  1453.9±125.83µs        ? ?/sec
   physical_plan_clickbench_q10                          1.00      2.2±0.18ms        ? ?/sec     1.03      2.2±0.13ms        ? ?/sec
   physical_plan_clickbench_q11                          1.02      2.4±0.18ms        ? ?/sec     1.00      2.3±0.19ms        ? ?/sec
   physical_plan_clickbench_q12                          1.00      2.4±0.21ms        ? ?/sec     1.10      2.6±0.13ms        ? ?/sec
   physical_plan_clickbench_q13                          1.01      2.3±0.15ms        ? ?/sec     1.00      2.3±0.18ms        ? ?/sec
   physical_plan_clickbench_q14                          1.00      2.3±0.24ms        ? ?/sec     1.08      2.4±0.23ms        ? ?/sec
   physical_plan_clickbench_q15                          1.00      2.4±0.15ms        ? ?/sec     1.03      2.4±0.15ms        ? ?/sec
   physical_plan_clickbench_q16                          1.00  1774.4±150.70µs        ? ?/sec    1.05  1860.8±179.56µs        ? ?/sec
   physical_plan_clickbench_q17                          1.04      2.0±0.13ms        ? ?/sec     1.00  1933.3±175.72µs        ? ?/sec
   physical_plan_clickbench_q18                          1.02  1782.1±175.69µs        ? ?/sec    1.00  1750.3±144.75µs        ? ?/sec
   physical_plan_clickbench_q19                          1.00      2.3±0.15ms        ? ?/sec     1.03      2.4±0.13ms        ? ?/sec
   physical_plan_clickbench_q2                           1.06      2.0±0.15ms        ? ?/sec     1.00  1916.1±142.65µs        ? ?/sec
   physical_plan_clickbench_q20                          1.00  1682.6±125.54µs        ? ?/sec    1.00  1683.6±136.44µs        ? ?/sec
   physical_plan_clickbench_q21                          1.00  1916.1±172.65µs        ? ?/sec    1.02  1950.8±189.00µs        ? ?/sec
   physical_plan_clickbench_q22                          1.00      2.4±0.18ms        ? ?/sec     1.08      2.6±0.13ms        ? ?/sec
   physical_plan_clickbench_q23                          1.00      2.5±0.24ms        ? ?/sec     1.03      2.6±0.19ms        ? ?/sec
   physical_plan_clickbench_q24                          1.00      5.3±0.13ms        ? ?/sec     1.04      5.5±0.10ms        ? ?/sec
   physical_plan_clickbench_q25                          1.00  1891.0±110.55µs        ? ?/sec    1.23      2.3±0.16ms        ? ?/sec
   physical_plan_clickbench_q26                          1.00  1828.1±86.94µs        ? ?/sec     1.17      2.1±0.17ms        ? ?/sec
   physical_plan_clickbench_q27                          1.00      2.0±0.13ms        ? ?/sec     1.15      2.3±0.13ms        ? ?/sec
   physical_plan_clickbench_q28                          1.00      2.4±0.17ms        ? ?/sec     1.20      2.8±0.19ms        ? ?/sec
   physical_plan_clickbench_q29                          1.00      2.5±0.20ms        ? ?/sec     1.16      2.9±0.13ms        ? ?/sec
   physical_plan_clickbench_q3                           1.02  1703.8±142.06µs        ? ?/sec    1.00  1664.9±130.39µs        ? ?/sec
   physical_plan_clickbench_q30                          1.00     13.0±0.20ms        ? ?/sec     1.00     13.0±0.08ms        ? ?/sec
   physical_plan_clickbench_q31                          1.00      2.7±0.12ms        ? ?/sec     1.06      2.9±0.26ms        ? ?/sec
   physical_plan_clickbench_q32                          1.00      2.8±0.15ms        ? ?/sec     1.08      3.0±0.12ms        ? ?/sec
   physical_plan_clickbench_q33                          1.00      2.3±0.17ms        ? ?/sec     1.12      2.5±0.09ms        ? ?/sec
   physical_plan_clickbench_q34                          1.00  1964.3±177.65µs        ? ?/sec    1.20      2.4±0.07ms        ? ?/sec
   physical_plan_clickbench_q35                          1.00  1963.2±193.75µs        ? ?/sec    1.24      2.4±0.07ms        ? ?/sec
   physical_plan_clickbench_q36                          1.11      2.4±0.20ms        ? ?/sec     1.00      2.2±0.17ms        ? ?/sec
   physical_plan_clickbench_q37                          1.11      2.8±0.21ms        ? ?/sec     1.00      2.5±0.12ms        ? ?/sec
   physical_plan_clickbench_q38                          1.12      3.0±0.12ms        ? ?/sec     1.00      2.7±0.12ms        ? ?/sec
   physical_plan_clickbench_q39                          1.11      3.0±0.17ms        ? ?/sec     1.00      2.7±0.15ms        ? ?/sec
   physical_plan_clickbench_q4                           1.00  1545.5±122.98µs        ? ?/sec    1.01  1555.3±106.02µs        ? ?/sec
   physical_plan_clickbench_q40                          1.02      3.6±0.24ms        ? ?/sec     1.00      3.5±0.16ms        ? ?/sec
   physical_plan_clickbench_q41                          1.04      3.3±0.20ms        ? ?/sec     1.00      3.2±0.18ms        ? ?/sec
   physical_plan_clickbench_q42                          1.00      3.3±0.16ms        ? ?/sec     1.00      3.3±0.12ms        ? ?/sec
   physical_plan_clickbench_q43                          1.06      3.5±0.20ms        ? ?/sec     1.00      3.3±0.14ms        ? ?/sec
   physical_plan_clickbench_q44                          1.07  1661.2±148.35µs        ? ?/sec    1.00  1548.1±66.05µs        ? ?/sec
   physical_plan_clickbench_q45                          1.06  1636.1±137.47µs        ? ?/sec    1.00  1540.8±61.29µs        ? ?/sec
   physical_plan_clickbench_q46                          1.03  1950.2±145.01µs        ? ?/sec    1.00  1887.6±76.16µs        ? ?/sec
   physical_plan_clickbench_q47                          1.11      2.8±0.22ms        ? ?/sec     1.00      2.5±0.15ms        ? ?/sec
   physical_plan_clickbench_q48                          1.03      2.8±0.26ms        ? ?/sec     1.00      2.7±0.16ms        ? ?/sec
   physical_plan_clickbench_q49                          1.07      3.0±0.18ms        ? ?/sec     1.00      2.8±0.14ms        ? ?/sec
   physical_plan_clickbench_q5                           1.02  1688.9±155.08µs        ? ?/sec    1.00  1649.7±123.14µs        ? ?/sec
   physical_plan_clickbench_q50                          1.05      3.1±0.14ms        ? ?/sec     1.00      3.0±0.13ms        ? ?/sec
   physical_plan_clickbench_q51                          1.09      2.1±0.18ms        ? ?/sec     1.00  1958.8±84.71µs        ? ?/sec
   physical_plan_clickbench_q52                          1.04      2.6±0.14ms        ? ?/sec     1.00      2.5±0.15ms        ? ?/sec
   physical_plan_clickbench_q53                          1.13      2.0±0.18ms        ? ?/sec     1.00  1800.9±71.02µs        ? ?/sec
   physical_plan_clickbench_q54                          1.06  1866.5±183.83µs        ? ?/sec    1.00  1753.5±66.86µs        ? ?/sec
   physical_plan_clickbench_q55                          1.06  1750.0±180.27µs        ? ?/sec    1.00  1646.9±57.41µs        ? ?/sec
   physical_plan_clickbench_q56                          1.04  1804.6±175.90µs        ? ?/sec    1.00  1738.3±59.00µs        ? ?/sec
   physical_plan_clickbench_q57                          1.10      2.0±0.19ms        ? ?/sec     1.00  1831.8±68.96µs        ? ?/sec
   physical_plan_clickbench_q58                          1.05      2.5±0.25ms        ? ?/sec     1.00      2.4±0.09ms        ? ?/sec
   physical_plan_clickbench_q59                          1.14  1920.2±209.57µs        ? ?/sec    1.00  1689.5±54.47µs        ? ?/sec
   physical_plan_clickbench_q6                           1.05  1694.9±135.29µs        ? ?/sec    1.00  1607.5±123.70µs        ? ?/sec
   physical_plan_clickbench_q60                          1.05  1824.0±165.86µs        ? ?/sec    1.00  1734.2±76.78µs        ? ?/sec
   physical_plan_clickbench_q7                           1.02  1501.2±87.10µs        ? ?/sec     1.00  1477.8±85.55µs        ? ?/sec
   physical_plan_clickbench_q8                           1.05      2.1±0.15ms        ? ?/sec     1.00  1966.3±157.97µs        ? ?/sec
   physical_plan_clickbench_q9                           1.00      2.0±0.16ms        ? ?/sec     1.04      2.1±0.16ms        ? ?/sec
   physical_plan_struct_join_agg_sort                    1.00   1182.7±3.49µs        ? ?/sec     1.08  1280.0±35.54µs        ? ?/sec
   physical_plan_tpcds_all                               1.00    603.4±9.46ms        ? ?/sec     1.00    604.3±9.97ms        ? ?/sec
   physical_plan_tpch_all                                1.00     43.4±1.65ms        ? ?/sec     1.00     43.4±1.60ms        ? ?/sec
   physical_plan_tpch_q1                                 1.00   1368.7±3.57µs        ? ?/sec     1.02  1401.0±27.12µs        ? ?/sec
   physical_plan_tpch_q10                                1.00      2.1±0.01ms        ? ?/sec     1.09      2.3±0.07ms        ? ?/sec
   physical_plan_tpch_q11                                1.00      2.0±0.01ms        ? ?/sec     1.12      2.3±0.12ms        ? ?/sec
   physical_plan_tpch_q12                                1.00   1187.7±4.53µs        ? ?/sec     1.08  1278.2±48.14µs        ? ?/sec
   physical_plan_tpch_q13                                1.00    955.6±6.35µs        ? ?/sec     1.02   972.1±14.05µs        ? ?/sec
   physical_plan_tpch_q14                                1.00  1238.8±12.34µs        ? ?/sec     1.05  1298.1±30.13µs        ? ?/sec
   physical_plan_tpch_q16                                1.00  1556.2±16.85µs        ? ?/sec     1.03  1601.3±48.20µs        ? ?/sec
   physical_plan_tpch_q17                                1.00  1563.9±32.25µs        ? ?/sec     1.03  1604.7±57.53µs        ? ?/sec
   physical_plan_tpch_q18                                1.00  1796.3±47.55µs        ? ?/sec     1.05  1888.8±77.90µs        ? ?/sec
   physical_plan_tpch_q19                                1.00      2.1±0.05ms        ? ?/sec     1.03      2.1±0.09ms        ? ?/sec
   physical_plan_tpch_q2                                 1.00      3.2±0.02ms        ? ?/sec     1.08      3.5±0.10ms        ? ?/sec
   physical_plan_tpch_q20                                1.00      2.1±0.08ms        ? ?/sec     1.02      2.1±0.11ms        ? ?/sec
   physical_plan_tpch_q21                                1.03      2.7±0.08ms        ? ?/sec     1.00      2.6±0.09ms        ? ?/sec
   physical_plan_tpch_q22                                1.03  1571.6±65.49µs        ? ?/sec     1.00  1527.4±54.18µs        ? ?/sec
   physical_plan_tpch_q3                                 1.00   1590.4±5.61µs        ? ?/sec     1.08  1717.0±65.55µs        ? ?/sec
   physical_plan_tpch_q4                                 1.00   1043.9±3.43µs        ? ?/sec     1.02  1064.1±22.65µs        ? ?/sec
   physical_plan_tpch_q5                                 1.00      2.1±0.01ms        ? ?/sec     1.07      2.3±0.09ms        ? ?/sec
   physical_plan_tpch_q6                                 1.00    578.9±1.89µs        ? ?/sec     1.01    586.1±6.64µs        ? ?/sec
   physical_plan_tpch_q7                                 1.00      2.5±0.02ms        ? ?/sec     1.08      2.7±0.12ms        ? ?/sec
   physical_plan_tpch_q8                                 1.00      3.3±0.03ms        ? ?/sec     1.10      3.6±0.10ms        ? ?/sec
   physical_plan_tpch_q9                                 1.00      2.5±0.01ms        ? ?/sec     1.06      2.6±0.11ms        ? ?/sec
   physical_select_aggregates_from_200                   1.00      7.5±0.04ms        ? ?/sec     1.02      7.7±0.04ms        ? ?/sec
   physical_select_all_from_1000                         1.00     17.0±0.07ms        ? ?/sec     1.01     17.1±0.08ms        ? ?/sec
   physical_select_one_from_700                          1.00    632.6±3.08µs        ? ?/sec     1.02    646.7±3.87µs        ? ?/sec
   physical_sorted_union_order_by_10_int64               1.00      3.5±0.01ms        ? ?/sec     1.05      3.7±0.08ms        ? ?/sec
   physical_sorted_union_order_by_10_uint64              1.00      6.8±0.05ms        ? ?/sec     1.03      7.1±0.08ms        ? ?/sec
   physical_sorted_union_order_by_50_int64               1.00     79.4±0.34ms        ? ?/sec     1.03     82.0±0.99ms        ? ?/sec
   physical_sorted_union_order_by_50_uint64              1.00    264.6±1.72ms        ? ?/sec     1.02    269.3±2.56ms        ? ?/sec
   physical_theta_join_consider_sort                     1.00    985.3±1.80µs        ? ?/sec     1.02   1001.7±8.27µs        ? ?/sec
   physical_unnest_to_join                               1.00    545.4±1.95µs        ? ?/sec     1.00    544.8±3.75µs        ? ?/sec
   physical_window_function_partition_by_12_on_values    1.00    629.9±1.79µs        ? ?/sec     1.00    632.3±6.08µs        ? ?/sec
   physical_window_function_partition_by_30_on_values    1.00   1187.8±3.71µs        ? ?/sec     1.01   1198.0±8.57µs        ? ?/sec
   physical_window_function_partition_by_4_on_values     1.00    385.2±0.78µs        ? ?/sec     1.00    385.5±2.36µs        ? ?/sec
   physical_window_function_partition_by_7_on_values     1.00    476.5±1.69µs        ? ?/sec     1.02    485.0±2.01µs        ? ?/sec
   physical_window_function_partition_by_8_on_values     1.00    505.5±1.50µs        ? ?/sec     1.00    504.7±2.69µs        ? ?/sec
   with_param_values_many_columns                        1.00    370.8±2.38µs        ? ?/sec     1.00    371.3±1.91µs        ? ?/sec
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **sql_planner — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 2015.4s |
   | Peak memory | 142.3 MiB |
   | Avg memory | 91.6 MiB |
   | CPU user | 1948.7s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **sql_planner — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 2055.5s |
   | Peak memory | 141.3 MiB |
   | Avg memory | 89.4 MiB |
   | CPU user | 1945.8s |
   | CPU sys | 1.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6245135045

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26461?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `94.91525%` with `3 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.90%. Comparing base ([`290ad15`](https://app.codecov.io/gh/apache/datafusion/commit/290ad15cc842167ca4d3c0774ccdf3a1d9d61862?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2689cbb`](https://app.codecov.io/gh/apache/datafusion/commit/2689cbbfae6c63ce9073b0a95cf09c4328fc75d7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26461?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/26461?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 91.89% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26461?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@           Coverage Diff           @@
   ##             main   #26461   +/-   ##
   =======================================
     Coverage   82.89%   82.90%           
   =======================================
     Files        1154     1154           
     Lines      457656   457711   +55     
     Branches   457656   457711   +55     
   =======================================
   + Hits       379391   379450   +59     
   + Misses      55261    55252    -9     
   - Partials    23004    23009    +5     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26461?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6250436998

   run benchmark sql_planner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6268896458

   @alamb Thanks for running these. The second run is the quieter one. In it, `physical_plan_tpch_all` is 1.02, `physical_plan_tpcds_all` is 1.01 and `physical_plan_clickbench_all` is 1.00. In the first run, per-query numbers move by up to 10% in both directions, which is more than the gap between the branches.
   
   The change should not add planning work. Before, the planner walked the final plan twice: once in `SanityCheckPlan` and once in its own `InvariantChecker(Executable)` pass. Now it walks the plan once and runs the same checks. The only new cost when planning succeeds is one `Arc` clone per rule, about 20 in the default list. The extra search that finds which rule to name runs only when the check fails.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4314641544


##########
datafusion/physical-optimizer/src/optimizer.rs:
##########
@@ -196,15 +195,10 @@ impl PhysicalOptimizer {
             // Therefore, it should be run at the end of the optimization process since any changes to the plan may break the dynamic filter's references.
             // See `FilterPushdownPhase` for more details.
             Arc::new(FilterPushdown::new_post_optimization()),
-            // The SanityCheckPlan rule checks whether the order and
-            // distribution requirements of each node in the plan
-            // is satisfied. It will also reject non-runnable query
-            // plans that use pipeline-breaking operators on infinite
-            // input(s). The rule generates a diagnostic error
-            // message for invalid plans. It makes no changes to the
-            // given query plan; i.e. it only acts as a final
-            // gatekeeping rule.
-            Arc::new(SanityCheckPlan::new()),
+            // There is no `SanityCheckPlan` rule at the end of this list. The

Review Comment:
   Done. `SanityCheckPlan` is now deprecated since 56.0.0, and the note points to `check_plan_tree_sanity` for code that runs optimizer rules without the planner. Tests that only used the rule to validate a plan now call `check_plan_tree_sanity` directly, and the upgrade guide says the rule is deprecated.
   
   I deprecated it rather than removing it because several downstream projects construct it in their own rule lists, for example greptimedb, cube, sail and openobserve. They should see a deprecation warning before they hit a compile error.
   



##########
datafusion/core/src/optimizer_rule_reference.md:
##########
@@ -69,26 +69,30 @@ Rule order matters. The default pipeline may change between releases.
 The same rule name may appear more than once when the default pipeline runs it
 in multiple phases.
 
-| order | rule                           | phase                   | summary                                                                                                      |
-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
-| 1     | `OutputRequirements`           | add phase               | Adds helper nodes so output requirements survive later physical rewrites.                                    |
-| 2     | `aggregate_statistics`         | -                       | Uses exact source statistics to answer some aggregates without scanning data.                                |
-| 3     | `join_selection`               | -                       | Chooses join implementation, build side, and partition mode from statistics and stream properties.           |
-| 4     | `LimitedDistinctAggregation`   | -                       | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed.              |
-| 5     | `FilterPushdown`               | pre-optimization phase  | Pushes supported physical filters down toward data sources before distribution and sorting are enforced.     |
-| 6     | `WindowTopN`                   | -                       | Replaces eligible row-number window and filter patterns with per-partition TopK execution.                   |
-| 7     | `EnsureRequirements`           | -                       | Enforces both distribution and sorting requirements in a single idempotent rule.                             |
-| 8     | `CombinePartialFinalAggregate` | -                       | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.             |
-| 9     | `OptimizeAggregateOrder`       | -                       | Updates aggregate expressions to use the best ordering once sort requirements are known.                     |
-| 10    | `ProjectionPushdown`           | early pass              | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure.           |
-| 11    | `OutputRequirements`           | remove phase            | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done.          |
-| 12    | `LimitAggregation`             | -                       | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets.                   |
-| 13    | `LimitPushPastWindows`         | -                       | Pushes fetch limits through bounded window operators when doing so keeps the result correct.                 |
-| 14    | `HashJoinBuffering`            | -                       | Adds buffering on the probe side of hash joins so probing can start before build completion.                 |
-| 15    | `LimitPushdown`                | -                       | Moves physical limits into child operators or fetch-enabled variants to cut data early.                      |
-| 16    | `TopKRepartition`              | -                       | Pushes TopK below hash repartition when the partition key is a prefix of the sort key.                       |
-| 17    | `ProjectionPushdown`           | late pass               | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities.               |
-| 18    | `PushdownSort`                 | -                       | Pushes sort requirements into data sources that can already return sorted output.                            |
-| 19    | `EnsureCooperative`            | -                       | Wraps non-cooperative plan parts so long-running tasks yield fairly.                                         |
-| 20    | `FilterPushdown(Post)`         | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving.                        |
-| 21    | `SanityCheckPlan`              | -                       | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
+After the last rule, including any rule added by the user, the physical planner

Review Comment:
   Agreed. #26351 has merged, so a follow-up PR can move this list next to the contract docs.
   
   This PR is now rebased on it. I also removed `SanityCheckPlan` from the contract's list of rules that must keep a fixed order, and added a sentence saying the planner checks the plan after the last rule.
   



##########
datafusion/core/src/optimizer_rule_reference.md:
##########
@@ -69,26 +69,30 @@ Rule order matters. The default pipeline may change between releases.
 The same rule name may appear more than once when the default pipeline runs it
 in multiple phases.
 
-| order | rule                           | phase                   | summary                                                                                                      |
-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
-| 1     | `OutputRequirements`           | add phase               | Adds helper nodes so output requirements survive later physical rewrites.                                    |
-| 2     | `aggregate_statistics`         | -                       | Uses exact source statistics to answer some aggregates without scanning data.                                |
-| 3     | `join_selection`               | -                       | Chooses join implementation, build side, and partition mode from statistics and stream properties.           |
-| 4     | `LimitedDistinctAggregation`   | -                       | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed.              |
-| 5     | `FilterPushdown`               | pre-optimization phase  | Pushes supported physical filters down toward data sources before distribution and sorting are enforced.     |
-| 6     | `WindowTopN`                   | -                       | Replaces eligible row-number window and filter patterns with per-partition TopK execution.                   |
-| 7     | `EnsureRequirements`           | -                       | Enforces both distribution and sorting requirements in a single idempotent rule.                             |
-| 8     | `CombinePartialFinalAggregate` | -                       | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.             |
-| 9     | `OptimizeAggregateOrder`       | -                       | Updates aggregate expressions to use the best ordering once sort requirements are known.                     |
-| 10    | `ProjectionPushdown`           | early pass              | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure.           |
-| 11    | `OutputRequirements`           | remove phase            | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done.          |
-| 12    | `LimitAggregation`             | -                       | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets.                   |
-| 13    | `LimitPushPastWindows`         | -                       | Pushes fetch limits through bounded window operators when doing so keeps the result correct.                 |
-| 14    | `HashJoinBuffering`            | -                       | Adds buffering on the probe side of hash joins so probing can start before build completion.                 |
-| 15    | `LimitPushdown`                | -                       | Moves physical limits into child operators or fetch-enabled variants to cut data early.                      |
-| 16    | `TopKRepartition`              | -                       | Pushes TopK below hash repartition when the partition key is a prefix of the sort key.                       |
-| 17    | `ProjectionPushdown`           | late pass               | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities.               |
-| 18    | `PushdownSort`                 | -                       | Pushes sort requirements into data sources that can already return sorted output.                            |
-| 19    | `EnsureCooperative`            | -                       | Wraps non-cooperative plan parts so long-running tasks yield fairly.                                         |
-| 20    | `FilterPushdown(Post)`         | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving.                        |
-| 21    | `SanityCheckPlan`              | -                       | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
+After the last rule, including any rule added by the user, the physical planner

Review Comment:
   > TIL about this document 🤯
   
   +1 😆 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub) via github" <gi...@datafusion.apache.org>.
jayzhan211 merged PR #26461:
URL: https://github.com/apache/datafusion/pull/26461


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6273166324

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion v55.1.0 (current)
          Built [  45.593s] (current)
        Parsing datafusion v55.1.0 (current)
         Parsed [   0.025s] (current)
       Building datafusion v55.1.0 (baseline)
          Built [  41.124s] (baseline)
        Parsing datafusion v55.1.0 (baseline)
         Parsed [   0.024s] (baseline)
       Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.479s] 229 checks: 229 pass, 31 skip
        Summary no semver update required
       Finished [  89.437s] datafusion
       Building datafusion-physical-optimizer v55.1.0 (current)
          Built [  27.971s] (current)
        Parsing datafusion-physical-optimizer v55.1.0 (current)
         Parsed [   0.015s] (current)
       Building datafusion-physical-optimizer v55.1.0 (baseline)
          Built [  28.716s] (baseline)
        Parsing datafusion-physical-optimizer v55.1.0 (baseline)
         Parsed [   0.016s] (baseline)
       Checking datafusion-physical-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.090s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip
   
   --- failure type_marked_deprecated: #[deprecated] added on type ---
   
   Description:
   A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
           ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_marked_deprecated.ron
   
   Failed in:
     Struct SanityCheckPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-optimizer/src/sanity_checker.rs:61
   
        Summary semver requires new minor version: 0 major and 1 minor checks failed
       Finished [  58.093s] datafusion-physical-optimizer
       Building datafusion-sqllogictest v55.1.0 (current)
          Built [  66.550s] (current)
        Parsing datafusion-sqllogictest v55.1.0 (current)
         Parsed [   0.013s] (current)
       Building datafusion-sqllogictest v55.1.0 (baseline)
          Built [  65.632s] (baseline)
        Parsing datafusion-sqllogictest v55.1.0 (baseline)
         Parsed [   0.013s] (baseline)
       Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.078s] 229 checks: 229 pass, 31 skip
        Summary no semver update required
       Finished [ 159.694s] datafusion-sqllogictest
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub) via github" <gi...@datafusion.apache.org>.
jayzhan211 commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6537244786

   Thanks @alamb @zhuqi-lucas for review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6246259013

   Thank you, I have a quick look and I think its overall beneficial. Will have a in-depth review after this PR getting some other +1s.
   
   Specifically, I think the benefits is prevent internal tests or simple extensions from building invalid optimizer pipeline; but for more advanced extension (that overrides sanity check), we might need additional API to set a custom sanity checker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6250437807

   
   run benchmark sql_planner
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4390035358


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3596,6 +3608,69 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {
     }
 }
 
+/// A physical optimizer rule and the plan it produced.
+type PlanAfterRule<'a> = (
+    &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,
+    Arc<dyn ExecutionPlan>,
+);
+
+/// Adds context to an error from the final [`check_plan_tree_sanity`].
+///
+/// `EnsureRequirements` makes the order and distribution requirements hold,
+/// and every rule after it must keep them. When they do not hold for
+/// `final_plan`, the context names the rule that broke this contract: either
+/// `EnsureRequirements` itself, or the rule after which the requirements
+/// stopped holding.
+fn sanity_check_error(
+    error: DataFusionError,
+    final_plan: &Arc<dyn ExecutionPlan>,
+    input_plan: &Arc<dyn ExecutionPlan>,
+    plan_after_rule: &[PlanAfterRule],
+) -> DataFusionError {
+    const SANITY_CHECK: &str = "SanityCheckPlan";
+    if input_requirements_hold(final_plan) {
+        // The error is about unbounded inputs or another invariant.
+        return DataFusionError::Context(SANITY_CHECK.to_string(), Box::new(error));
+    }
+
+    let ensure_requirements = EnsureRequirements::new();
+    let enforced = plan_after_rule
+        .iter()
+        .rposition(|(rule, _)| rule.name() == ensure_requirements.name());

Review Comment:
   Once #25688 lands, this search could start at the enforcement phase boundary instead of matching `EnsureRequirements` by name.



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3344,12 +3352,16 @@ impl DefaultPhysicalPlanner {
                 optimizer.name(),
                 displayable(new_plan.as_ref()).indent(false)
             );
-            observer(new_plan.as_ref(), optimizer.as_ref())
+            observer(new_plan.as_ref(), optimizer.as_ref());
+            plan_after_rule.push((optimizer, Arc::clone(&new_plan)));
         }
 
-        // This runs once after all optimizer runs are complete,
-        // to verify that the plan is executable.
-        InvariantChecker(InvariantLevel::Executable).check(&new_plan)?;
+        // This runs once after all optimizer runs are complete, including
+        // rules appended by the user, to verify that the plan is executable.
+        let optimizer_options = &session_state.config_options().optimizer;
+        if let Err(e) = check_plan_tree_sanity(&new_plan, optimizer_options) {

Review Comment:
   Since this check can no longer be skipped, an opt-out flag could be a follow-up if a downstream needs it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6250458703

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6250436998)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6250436998-3565-wf6gk 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing planner-final-plan-check (2689cbbfae6c63ce9073b0a95cf09c4328fc75d7) to 290ad15 (merge-base) [diff](https://github.com/apache/datafusion/compare/290ad15cc842167ca4d3c0774ccdf3a1d9d61862..2689cbbfae6c63ce9073b0a95cf09c4328fc75d7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6251166440

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6250436998)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing planner-final-plan-check (2689cbbfae6c63ce9073b0a95cf09c4328fc75d7) to 290ad15 (merge-base) [diff](https://github.com/apache/datafusion/compare/290ad15cc842167ca4d3c0774ccdf3a1d9d61862..2689cbbfae6c63ce9073b0a95cf09c4328fc75d7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   group                                                 HEAD                                   planner-final-plan-check
   -----                                                 ----                                   ------------------------
   logical_aggregate_with_join                           1.00    344.5±1.66µs        ? ?/sec    1.01    349.3±1.83µs        ? ?/sec
   logical_correlated_subquery_exists                    1.00    205.9±0.73µs        ? ?/sec    1.02    209.4±0.67µs        ? ?/sec
   logical_correlated_subquery_in                        1.00    209.0±1.40µs        ? ?/sec    1.02    213.7±1.68µs        ? ?/sec
   logical_distinct_many_columns                         1.00    290.5±0.90µs        ? ?/sec    1.01    294.5±2.17µs        ? ?/sec
   logical_join_4_with_agg_and_filter                    1.00    196.4±0.65µs        ? ?/sec    1.03    202.5±1.27µs        ? ?/sec
   logical_join_8_with_agg_sort_limit                    1.00    342.5±1.31µs        ? ?/sec    1.02    350.2±1.11µs        ? ?/sec
   logical_join_chain_16                                 1.00    619.2±2.33µs        ? ?/sec    1.01    627.1±2.14µs        ? ?/sec
   logical_join_chain_4                                  1.00     77.4±0.44µs        ? ?/sec    1.02     79.2±0.39µs        ? ?/sec
   logical_join_chain_8                                  1.00    201.3±0.56µs        ? ?/sec    1.03    207.1±1.71µs        ? ?/sec
   logical_multiple_subqueries                           1.00    393.5±1.41µs        ? ?/sec    1.01    398.4±2.00µs        ? ?/sec
   logical_nested_cte_4_levels                           1.00    200.0±0.56µs        ? ?/sec    1.00    200.3±0.71µs        ? ?/sec
   logical_plan_struct_join_agg_sort                     1.00    126.6±0.90µs        ? ?/sec    1.06    134.1±0.91µs        ? ?/sec
   logical_plan_tpcds_all                                1.00     71.6±0.13ms        ? ?/sec    1.01     72.2±0.12ms        ? ?/sec
   logical_plan_tpch_all                                 1.00      5.2±0.02ms        ? ?/sec    1.02      5.3±0.01ms        ? ?/sec
   logical_scalar_subquery                               1.00    216.5±0.87µs        ? ?/sec    1.02    221.2±1.01µs        ? ?/sec
   logical_select_all_from_1000                          1.00      6.8±0.01ms        ? ?/sec    1.01      6.9±0.03ms        ? ?/sec
   logical_select_one_from_700                           1.00    225.6±1.65µs        ? ?/sec    1.02    230.3±1.73µs        ? ?/sec
   logical_trivial_join_high_numbered_columns            1.00    207.4±1.32µs        ? ?/sec    1.02    210.9±1.17µs        ? ?/sec
   logical_trivial_join_low_numbered_columns             1.00    196.8±1.41µs        ? ?/sec    1.01    199.1±1.15µs        ? ?/sec
   logical_union_4_branches                              1.00    310.6±1.31µs        ? ?/sec    1.01    315.1±1.52µs        ? ?/sec
   logical_union_8_branches                              1.00    632.6±7.17µs        ? ?/sec    1.01    640.1±9.22µs        ? ?/sec
   logical_wide_aggregate_1000_exprs                     1.01     41.6±0.17ms        ? ?/sec    1.00     41.3±0.15ms        ? ?/sec
   logical_wide_aggregate_100_exprs                      1.00   1537.9±3.49µs        ? ?/sec    1.00   1537.8±3.64µs        ? ?/sec
   logical_wide_case_50_exprs                            1.00   1695.6±5.93µs        ? ?/sec    1.01   1711.0±3.96µs        ? ?/sec
   logical_wide_filter_200_predicates                    1.00   1300.8±4.88µs        ? ?/sec    1.00   1305.7±5.53µs        ? ?/sec
   logical_wide_filter_50_predicates                     1.00    341.2±1.71µs        ? ?/sec    1.01    345.9±3.82µs        ? ?/sec
   optimizer_correlated_exists                           1.00    217.5±0.55µs        ? ?/sec    1.00    217.9±0.61µs        ? ?/sec
   optimizer_join_4_with_agg_filter                      1.00    391.3±0.90µs        ? ?/sec    1.00    393.2±1.07µs        ? ?/sec
   optimizer_join_chain_4                                1.00    154.4±0.28µs        ? ?/sec    1.01    155.4±0.26µs        ? ?/sec
   optimizer_join_chain_8                                1.00    528.6±1.21µs        ? ?/sec    1.01    531.3±0.97µs        ? ?/sec
   optimizer_select_all_from_1000                        1.00      6.7±0.02ms        ? ?/sec    1.00      6.8±0.02ms        ? ?/sec
   optimizer_select_one_from_700                         1.00    235.8±0.78µs        ? ?/sec    1.03    242.7±0.95µs        ? ?/sec
   optimizer_tpcds_all                                   1.00    247.9±0.62ms        ? ?/sec    1.00    246.9±0.21ms        ? ?/sec
   optimizer_tpch_all                                    1.00     14.9±0.04ms        ? ?/sec    1.00     14.9±0.03ms        ? ?/sec
   optimizer_wide_aggregate_100                          1.00   1739.3±2.69µs        ? ?/sec    1.00   1741.7±1.56µs        ? ?/sec
   optimizer_wide_filter_200                             1.00      4.0±0.01ms        ? ?/sec    1.00      4.0±0.01ms        ? ?/sec
   physical_intersection                                 1.00    507.6±2.38µs        ? ?/sec    1.01    512.2±2.83µs        ? ?/sec
   physical_join_consider_sort                           1.00    951.2±1.99µs        ? ?/sec    1.00    950.9±2.17µs        ? ?/sec
   physical_join_distinct                                1.00    189.7±1.29µs        ? ?/sec    1.02    192.6±1.10µs        ? ?/sec
   physical_many_self_joins                              1.00      7.2±0.02ms        ? ?/sec    1.01      7.3±0.01ms        ? ?/sec
   physical_plan_clickbench_all                          1.00    128.6±0.54ms        ? ?/sec    1.00    128.8±0.29ms        ? ?/sec
   physical_plan_clickbench_q1                           1.00  1274.1±11.46µs        ? ?/sec    1.00   1273.4±8.45µs        ? ?/sec
   physical_plan_clickbench_q10                          1.00   1825.7±5.18µs        ? ?/sec    1.00   1816.7±5.52µs        ? ?/sec
   physical_plan_clickbench_q11                          1.00   1962.6±5.43µs        ? ?/sec    1.01  1973.2±11.47µs        ? ?/sec
   physical_plan_clickbench_q12                          1.00      2.1±0.01ms        ? ?/sec    1.01      2.1±0.02ms        ? ?/sec
   physical_plan_clickbench_q13                          1.00   1857.0±8.20µs        ? ?/sec    1.02  1896.4±37.69µs        ? ?/sec
   physical_plan_clickbench_q14                          1.00   1992.3±8.12µs        ? ?/sec    1.03      2.1±0.02ms        ? ?/sec
   physical_plan_clickbench_q15                          1.00   1908.5±4.99µs        ? ?/sec    1.02  1938.4±20.41µs        ? ?/sec
   physical_plan_clickbench_q16                          1.00  1640.6±11.74µs        ? ?/sec    1.00  1644.6±11.55µs        ? ?/sec
   physical_plan_clickbench_q17                          1.00   1684.3±5.62µs        ? ?/sec    1.01   1698.7±8.78µs        ? ?/sec
   physical_plan_clickbench_q18                          1.00   1523.7±4.88µs        ? ?/sec    1.02   1561.1±6.62µs        ? ?/sec
   physical_plan_clickbench_q19                          1.00   1872.5±4.70µs        ? ?/sec    1.04  1943.2±21.76µs        ? ?/sec
   physical_plan_clickbench_q2                           1.00   1623.6±4.55µs        ? ?/sec    1.00   1624.6±6.52µs        ? ?/sec
   physical_plan_clickbench_q20                          1.00   1434.8±4.73µs        ? ?/sec    1.03  1480.3±11.00µs        ? ?/sec
   physical_plan_clickbench_q21                          1.00   1629.2±5.73µs        ? ?/sec    1.00   1624.3±5.10µs        ? ?/sec
   physical_plan_clickbench_q22                          1.01   1973.3±8.68µs        ? ?/sec    1.00  1951.3±10.05µs        ? ?/sec
   physical_plan_clickbench_q23                          1.00      2.1±0.01ms        ? ?/sec    1.00      2.1±0.02ms        ? ?/sec
   physical_plan_clickbench_q24                          1.01      4.8±0.02ms        ? ?/sec    1.00      4.7±0.01ms        ? ?/sec
   physical_plan_clickbench_q25                          1.00   1723.3±8.18µs        ? ?/sec    1.01   1747.6±5.54µs        ? ?/sec
   physical_plan_clickbench_q26                          1.00   1612.9±8.31µs        ? ?/sec    1.00   1615.6±8.06µs        ? ?/sec
   physical_plan_clickbench_q27                          1.00   1766.2±5.63µs        ? ?/sec    1.01   1777.3±9.93µs        ? ?/sec
   physical_plan_clickbench_q28                          1.00      2.1±0.00ms        ? ?/sec    1.01      2.1±0.01ms        ? ?/sec
   physical_plan_clickbench_q29                          1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q3                           1.00   1519.1±5.28µs        ? ?/sec    1.00   1511.6±4.31µs        ? ?/sec
   physical_plan_clickbench_q30                          1.00     12.3±0.04ms        ? ?/sec    1.00     12.4±0.04ms        ? ?/sec
   physical_plan_clickbench_q31                          1.00      2.2±0.01ms        ? ?/sec    1.02      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q32                          1.00      2.2±0.01ms        ? ?/sec    1.01      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q33                          1.00   1812.1±4.89µs        ? ?/sec    1.03   1860.2±4.76µs        ? ?/sec
   physical_plan_clickbench_q34                          1.00   1654.3±4.00µs        ? ?/sec    1.00  1651.3±11.60µs        ? ?/sec
   physical_plan_clickbench_q35                          1.00   1677.0±4.16µs        ? ?/sec    1.02   1702.2±5.87µs        ? ?/sec
   physical_plan_clickbench_q36                          1.01   1910.1±4.23µs        ? ?/sec    1.00   1898.2±5.57µs        ? ?/sec
   physical_plan_clickbench_q37                          1.00      2.3±0.00ms        ? ?/sec    1.00      2.3±0.01ms        ? ?/sec
   physical_plan_clickbench_q38                          1.00      2.3±0.01ms        ? ?/sec    1.01      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q39                          1.00      2.4±0.02ms        ? ?/sec    1.01      2.5±0.01ms        ? ?/sec
   physical_plan_clickbench_q4                           1.02   1374.4±3.56µs        ? ?/sec    1.00   1348.0±4.01µs        ? ?/sec
   physical_plan_clickbench_q40                          1.03      3.1±0.02ms        ? ?/sec    1.00      3.0±0.01ms        ? ?/sec
   physical_plan_clickbench_q41                          1.00      2.7±0.02ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
   physical_plan_clickbench_q42                          1.01      2.7±0.02ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
   physical_plan_clickbench_q43                          1.01      2.9±0.05ms        ? ?/sec    1.00      2.9±0.01ms        ? ?/sec
   physical_plan_clickbench_q44                          1.02   1460.8±7.44µs        ? ?/sec    1.00   1432.4±6.25µs        ? ?/sec
   physical_plan_clickbench_q45                          1.02   1466.9±6.83µs        ? ?/sec    1.00   1440.4±3.53µs        ? ?/sec
   physical_plan_clickbench_q46                          1.00   1693.0±6.96µs        ? ?/sec    1.00   1691.5±7.59µs        ? ?/sec
   physical_plan_clickbench_q47                          1.01      2.3±0.01ms        ? ?/sec    1.00      2.3±0.01ms        ? ?/sec
   physical_plan_clickbench_q48                          1.00      2.4±0.01ms        ? ?/sec    1.01      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q49                          1.00      2.4±0.01ms        ? ?/sec    1.01      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q5                           1.00   1478.2±5.03µs        ? ?/sec    1.00   1478.5±4.42µs        ? ?/sec
   physical_plan_clickbench_q50                          1.00      2.5±0.01ms        ? ?/sec    1.02      2.6±0.01ms        ? ?/sec
   physical_plan_clickbench_q51                          1.00   1789.2±4.80µs        ? ?/sec    1.02  1823.3±19.13µs        ? ?/sec
   physical_plan_clickbench_q52                          1.01      2.3±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q53                          1.01   1641.8±7.83µs        ? ?/sec    1.00   1625.3±4.04µs        ? ?/sec
   physical_plan_clickbench_q54                          1.00   1653.2±9.93µs        ? ?/sec    1.00   1650.3±5.79µs        ? ?/sec
   physical_plan_clickbench_q55                          1.01   1583.1±9.11µs        ? ?/sec    1.00   1570.7±4.15µs        ? ?/sec
   physical_plan_clickbench_q56                          1.01  1588.7±13.20µs        ? ?/sec    1.00   1576.8±4.22µs        ? ?/sec
   physical_plan_clickbench_q57                          1.00   1694.5±6.52µs        ? ?/sec    1.00   1695.4±5.74µs        ? ?/sec
   physical_plan_clickbench_q58                          1.01      2.1±0.01ms        ? ?/sec    1.00      2.1±0.01ms        ? ?/sec
   physical_plan_clickbench_q59                          1.02   1601.9±6.11µs        ? ?/sec    1.00   1576.7±4.43µs        ? ?/sec
   physical_plan_clickbench_q6                           1.00   1465.9±6.46µs        ? ?/sec    1.01   1480.7±4.09µs        ? ?/sec
   physical_plan_clickbench_q60                          1.00   1576.7±6.15µs        ? ?/sec    1.01   1592.5±4.77µs        ? ?/sec
   physical_plan_clickbench_q7                           1.02   1351.0±4.24µs        ? ?/sec    1.00   1325.5±4.10µs        ? ?/sec
   physical_plan_clickbench_q8                           1.00   1817.5±8.32µs        ? ?/sec    1.01   1833.4±6.52µs        ? ?/sec
   physical_plan_clickbench_q9                           1.00   1747.2±4.77µs        ? ?/sec    1.01   1759.6±5.22µs        ? ?/sec
   physical_plan_struct_join_agg_sort                    1.00   1191.0±2.32µs        ? ?/sec    1.00   1189.8±7.65µs        ? ?/sec
   physical_plan_tpcds_all                               1.00    556.6±0.45ms        ? ?/sec    1.01    561.7±3.32ms        ? ?/sec
   physical_plan_tpch_all                                1.00     39.1±0.05ms        ? ?/sec    1.02     39.8±0.06ms        ? ?/sec
   physical_plan_tpch_q1                                 1.00   1361.2±1.99µs        ? ?/sec    1.01   1368.2±2.39µs        ? ?/sec
   physical_plan_tpch_q10                                1.00      2.1±0.00ms        ? ?/sec    1.02      2.1±0.00ms        ? ?/sec
   physical_plan_tpch_q11                                1.00      2.0±0.00ms        ? ?/sec    1.02      2.1±0.00ms        ? ?/sec
   physical_plan_tpch_q12                                1.00   1196.2±1.75µs        ? ?/sec    1.01  1211.3±13.07µs        ? ?/sec
   physical_plan_tpch_q13                                1.00    954.6±5.16µs        ? ?/sec    1.01    967.5±2.38µs        ? ?/sec
   physical_plan_tpch_q14                                1.00  1251.7±10.12µs        ? ?/sec    1.00   1247.9±2.02µs        ? ?/sec
   physical_plan_tpch_q16                                1.00   1507.9±3.04µs        ? ?/sec    1.02  1530.6±10.10µs        ? ?/sec
   physical_plan_tpch_q17                                1.00   1509.7±3.87µs        ? ?/sec    1.01   1528.0±2.76µs        ? ?/sec
   physical_plan_tpch_q18                                1.00   1696.9±3.16µs        ? ?/sec    1.02   1735.4±4.34µs        ? ?/sec
   physical_plan_tpch_q19                                1.00   1959.2±2.66µs        ? ?/sec    1.00   1958.4±2.79µs        ? ?/sec
   physical_plan_tpch_q2                                 1.00      3.2±0.00ms        ? ?/sec    1.00      3.2±0.00ms        ? ?/sec
   physical_plan_tpch_q20                                1.00   1940.1±2.28µs        ? ?/sec    1.02   1971.3±4.06µs        ? ?/sec
   physical_plan_tpch_q21                                1.00      2.4±0.00ms        ? ?/sec    1.01      2.4±0.00ms        ? ?/sec
   physical_plan_tpch_q22                                1.00   1446.9±3.80µs        ? ?/sec    1.03   1487.9±3.08µs        ? ?/sec
   physical_plan_tpch_q3                                 1.00   1593.3±2.68µs        ? ?/sec    1.00   1601.2±2.14µs        ? ?/sec
   physical_plan_tpch_q4                                 1.00   1035.2±1.67µs        ? ?/sec    1.03   1062.7±2.46µs        ? ?/sec
   physical_plan_tpch_q5                                 1.00      2.1±0.01ms        ? ?/sec    1.00      2.1±0.00ms        ? ?/sec
   physical_plan_tpch_q6                                 1.00    587.6±1.38µs        ? ?/sec    1.01    592.5±1.58µs        ? ?/sec
   physical_plan_tpch_q7                                 1.00      2.4±0.00ms        ? ?/sec    1.01      2.5±0.00ms        ? ?/sec
   physical_plan_tpch_q8                                 1.00      3.3±0.00ms        ? ?/sec    1.01      3.3±0.00ms        ? ?/sec
   physical_plan_tpch_q9                                 1.00      2.4±0.00ms        ? ?/sec    1.01      2.5±0.00ms        ? ?/sec
   physical_select_aggregates_from_200                   1.00      7.4±0.01ms        ? ?/sec    1.00      7.4±0.01ms        ? ?/sec
   physical_select_all_from_1000                         1.00     16.8±0.04ms        ? ?/sec    1.01     16.9±0.08ms        ? ?/sec
   physical_select_one_from_700                          1.00    632.8±3.15µs        ? ?/sec    1.02    646.8±4.32µs        ? ?/sec
   physical_sorted_union_order_by_10_int64               1.00      3.5±0.00ms        ? ?/sec    1.00      3.5±0.00ms        ? ?/sec
   physical_sorted_union_order_by_10_uint64              1.00      6.7±0.01ms        ? ?/sec    1.00      6.7±0.01ms        ? ?/sec
   physical_sorted_union_order_by_50_int64               1.00     77.7±0.20ms        ? ?/sec    1.01     78.6±0.21ms        ? ?/sec
   physical_sorted_union_order_by_50_uint64              1.00    254.2±0.53ms        ? ?/sec    1.00    253.5±1.53ms        ? ?/sec
   physical_theta_join_consider_sort                     1.00    978.1±4.09µs        ? ?/sec    1.00    976.6±2.67µs        ? ?/sec
   physical_unnest_to_join                               1.00    549.2±2.07µs        ? ?/sec    1.00    547.0±1.36µs        ? ?/sec
   physical_window_function_partition_by_12_on_values    1.01    625.9±0.98µs        ? ?/sec    1.00    619.2±2.09µs        ? ?/sec
   physical_window_function_partition_by_30_on_values    1.02   1203.9±1.57µs        ? ?/sec    1.00   1184.8±2.07µs        ? ?/sec
   physical_window_function_partition_by_4_on_values     1.01    395.3±0.84µs        ? ?/sec    1.00    390.9±3.50µs        ? ?/sec
   physical_window_function_partition_by_7_on_values     1.01    483.8±0.84µs        ? ?/sec    1.00    478.3±0.90µs        ? ?/sec
   physical_window_function_partition_by_8_on_values     1.03    513.6±4.69µs        ? ?/sec    1.00    500.7±0.99µs        ? ?/sec
   with_param_values_many_columns                        1.00    367.3±1.41µs        ? ?/sec    1.01    371.2±1.62µs        ? ?/sec
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **sql_planner — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 2070.5s |
   | Peak memory | 137.6 MiB |
   | Avg memory | 87.5 MiB |
   | CPU user | 1977.0s |
   | CPU sys | 1.5s |
   | Peak spill | 0 B |
   
   **sql_planner — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 2055.5s |
   | Peak memory | 141.8 MiB |
   | Avg memory | 90.8 MiB |
   | CPU user | 1985.5s |
   | CPU sys | 1.5s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6507278150

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6507161755)
   **Instance:** `c4a-highmem-16-lssd` (12 vCPU / 65 GiB) | `Linux bench-c6507161755-4125-tr7jr 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing planner-final-plan-check (0909e8e5ca71b04ba3f12aef93a719597564918e) to e8940ac (merge-base) [diff](https://github.com/apache/datafusion/compare/e8940acd4ab0baa1d68599ac2aec453a0a6ece94..0909e8e5ca71b04ba3f12aef93a719597564918e)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6507161755

   run benchmark sql_planner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub) via github" <gi...@datafusion.apache.org>.
jayzhan211 commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4429781871


##########
datafusion/core/tests/memory_limit/mod.rs:
##########
@@ -578,7 +578,7 @@ async fn sort_merge_join_no_spill() {
         ])
         .with_memory_limit(1_000)
         .with_config(config)
-        .with_scenario(Scenario::AccessLogStreaming)
+        .with_scenario(Scenario::AccessLogStreamingByPodTime)

Review Comment:
   These two tests aren't checking optimizer errors. They still check the sort merge join's memory use. Their rule list is only `JoinSelection`, so the `SortMergeJoinExec` read unsorted input, and it ran only because that list left out `SanityCheckPlan`. The planner now rejects that plan, so the tests sort the generated rows by the join keys and declare that order on the table. That keeps the plan valid without adding a `SortExec`, whose memory budget would be hit before the join's. The optimizer error tests are in `core/tests/physical_optimizer/sanity_checker.rs`, so I'd keep these two here as they are.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub) via github" <gi...@datafusion.apache.org>.
alamb commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6557986377

   Thanks for pushing this along


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6508663379

   🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6507161755)
   
   **Instance:** `c4a-highmem-16-lssd` (12 vCPU / 65 GiB)
   
   Comparing planner-final-plan-check (0909e8e5ca71b04ba3f12aef93a719597564918e) to e8940ac (merge-base) [diff](https://github.com/apache/datafusion/compare/e8940acd4ab0baa1d68599ac2aec453a0a6ece94..0909e8e5ca71b04ba3f12aef93a719597564918e)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   group                                                 HEAD                                   planner-final-plan-check
   -----                                                 ----                                   ------------------------
   logical_aggregate_with_join                           1.00    350.1±1.53µs        ? ?/sec    1.01    354.8±7.84µs        ? ?/sec
   logical_correlated_subquery_exists                    1.01    207.5±1.32µs        ? ?/sec    1.00    205.3±0.90µs        ? ?/sec
   logical_correlated_subquery_in                        1.02    210.5±1.26µs        ? ?/sec    1.00    207.0±1.19µs        ? ?/sec
   logical_distinct_many_columns                         1.01    292.1±1.18µs        ? ?/sec    1.00    290.3±0.98µs        ? ?/sec
   logical_join_4_with_agg_and_filter                    1.00    194.5±1.25µs        ? ?/sec    1.01    196.9±1.22µs        ? ?/sec
   logical_join_8_with_agg_sort_limit                    1.00    340.1±2.03µs        ? ?/sec    1.00    339.4±1.88µs        ? ?/sec
   logical_join_chain_16                                 1.00    611.9±3.78µs        ? ?/sec    1.00    610.0±2.47µs        ? ?/sec
   logical_join_chain_4                                  1.00     73.1±0.41µs        ? ?/sec    1.01     74.1±0.34µs        ? ?/sec
   logical_join_chain_8                                  1.00    196.3±0.86µs        ? ?/sec    1.00    197.3±0.83µs        ? ?/sec
   logical_multiple_subqueries                           1.03    399.6±1.07µs        ? ?/sec    1.00    389.3±1.82µs        ? ?/sec
   logical_nested_cte_4_levels                           1.01    197.8±1.39µs        ? ?/sec    1.00    196.4±1.26µs        ? ?/sec
   logical_plan_struct_join_agg_sort                     1.00    125.2±0.56µs        ? ?/sec    1.02    127.3±1.06µs        ? ?/sec
   logical_plan_tpcds_all                                1.00     71.9±0.18ms        ? ?/sec    1.03     73.8±0.44ms        ? ?/sec
   logical_plan_tpch_all                                 1.00      5.3±0.02ms        ? ?/sec    1.03      5.4±0.03ms        ? ?/sec
   logical_scalar_subquery                               1.03    221.3±0.77µs        ? ?/sec    1.00    214.3±0.81µs        ? ?/sec
   logical_select_all_from_1000                          1.00      6.9±0.16ms        ? ?/sec    1.03      7.2±0.04ms        ? ?/sec
   logical_select_one_from_700                           1.03    236.6±2.32µs        ? ?/sec    1.00    228.7±1.97µs        ? ?/sec
   logical_trivial_join_high_numbered_columns            1.02    210.4±1.61µs        ? ?/sec    1.00    207.1±1.20µs        ? ?/sec
   logical_trivial_join_low_numbered_columns             1.01    198.8±1.27µs        ? ?/sec    1.00    197.7±1.32µs        ? ?/sec
   logical_union_4_branches                              1.01    312.0±1.84µs        ? ?/sec    1.00    308.1±2.86µs        ? ?/sec
   logical_union_8_branches                              1.00    627.2±1.85µs        ? ?/sec    1.00    628.3±7.19µs        ? ?/sec
   logical_wide_aggregate_1000_exprs                     1.00     38.2±0.11ms        ? ?/sec    1.02     38.8±0.22ms        ? ?/sec
   logical_wide_aggregate_100_exprs                      1.00   1567.1±4.20µs        ? ?/sec    1.00   1561.1±4.12µs        ? ?/sec
   logical_wide_case_50_exprs                            1.00   1708.4±5.85µs        ? ?/sec    1.00   1711.9±7.19µs        ? ?/sec
   logical_wide_filter_200_predicates                    1.00   1312.5±6.81µs        ? ?/sec    1.00   1313.5±5.41µs        ? ?/sec
   logical_wide_filter_50_predicates                     1.01    343.7±3.53µs        ? ?/sec    1.00    341.0±1.47µs        ? ?/sec
   optimizer_correlated_exists                           1.00    217.6±0.82µs        ? ?/sec    1.01    220.7±1.04µs        ? ?/sec
   optimizer_join_4_with_agg_filter                      1.00    396.2±1.34µs        ? ?/sec    1.01    400.7±1.58µs        ? ?/sec
   optimizer_join_chain_4                                1.00    153.6±0.38µs        ? ?/sec    1.01    155.1±0.47µs        ? ?/sec
   optimizer_join_chain_8                                1.00    533.5±1.21µs        ? ?/sec    1.01    537.5±0.81µs        ? ?/sec
   optimizer_select_all_from_1000                        1.00      6.8±0.04ms        ? ?/sec    1.09      7.4±0.01ms        ? ?/sec
   optimizer_select_one_from_700                         1.02    240.8±1.05µs        ? ?/sec    1.00    236.8±0.88µs        ? ?/sec
   optimizer_tpcds_all                                   1.00    247.1±0.24ms        ? ?/sec    1.01    250.5±0.81ms        ? ?/sec
   optimizer_tpch_all                                    1.00     14.9±0.03ms        ? ?/sec    1.03     15.3±0.06ms        ? ?/sec
   optimizer_wide_aggregate_100                          1.00   1750.2±2.58µs        ? ?/sec    1.00   1750.9±2.62µs        ? ?/sec
   optimizer_wide_filter_200                             1.00      4.0±0.01ms        ? ?/sec    1.03      4.1±0.01ms        ? ?/sec
   physical_intersection                                 1.02    516.5±2.24µs        ? ?/sec    1.00    506.3±2.29µs        ? ?/sec
   physical_join_consider_sort                           1.01    967.4±1.59µs        ? ?/sec    1.00    959.5±1.38µs        ? ?/sec
   physical_join_distinct                                1.01    193.5±1.34µs        ? ?/sec    1.00    190.7±1.54µs        ? ?/sec
   physical_many_self_joins                              1.00      7.3±0.01ms        ? ?/sec    1.00      7.3±0.04ms        ? ?/sec
   physical_plan_clickbench_all                          1.00    127.9±0.25ms        ? ?/sec    1.03    131.3±0.35ms        ? ?/sec
   physical_plan_clickbench_q1                           1.00   1273.7±8.63µs        ? ?/sec    1.01   1288.2±8.53µs        ? ?/sec
   physical_plan_clickbench_q10                          1.01   1867.2±5.50µs        ? ?/sec    1.00   1849.5±6.09µs        ? ?/sec
   physical_plan_clickbench_q11                          1.00   1980.9±8.02µs        ? ?/sec    1.02      2.0±0.01ms        ? ?/sec
   physical_plan_clickbench_q12                          1.07      2.2±0.16ms        ? ?/sec    1.00      2.1±0.01ms        ? ?/sec
   physical_plan_clickbench_q13                          1.11      2.1±0.07ms        ? ?/sec    1.00   1891.2±5.75µs        ? ?/sec
   physical_plan_clickbench_q14                          1.00   1995.2±7.38µs        ? ?/sec    1.00      2.0±0.00ms        ? ?/sec
   physical_plan_clickbench_q15                          1.01  1961.8±11.49µs        ? ?/sec    1.00   1942.9±6.34µs        ? ?/sec
   physical_plan_clickbench_q16                          1.00   1652.7±4.95µs        ? ?/sec    1.00   1660.4±5.48µs        ? ?/sec
   physical_plan_clickbench_q17                          1.00   1661.6±5.04µs        ? ?/sec    1.01   1685.0±5.52µs        ? ?/sec
   physical_plan_clickbench_q18                          1.01   1544.5±5.58µs        ? ?/sec    1.00   1524.9±4.66µs        ? ?/sec
   physical_plan_clickbench_q19                          1.00   1866.1±5.85µs        ? ?/sec    1.02   1895.6±7.68µs        ? ?/sec
   physical_plan_clickbench_q2                           1.00   1659.0±5.17µs        ? ?/sec    1.00   1660.3±5.69µs        ? ?/sec
   physical_plan_clickbench_q20                          1.01   1459.6±6.21µs        ? ?/sec    1.00   1445.3±6.61µs        ? ?/sec
   physical_plan_clickbench_q21                          1.01   1647.8±5.58µs        ? ?/sec    1.00   1628.7±5.61µs        ? ?/sec
   physical_plan_clickbench_q22                          1.00      2.0±0.00ms        ? ?/sec    1.00      2.0±0.01ms        ? ?/sec
   physical_plan_clickbench_q23                          1.00      2.1±0.01ms        ? ?/sec    1.02      2.1±0.02ms        ? ?/sec
   physical_plan_clickbench_q24                          1.00      4.8±0.03ms        ? ?/sec    1.00      4.8±0.01ms        ? ?/sec
   physical_plan_clickbench_q25                          1.00   1762.0±4.56µs        ? ?/sec    1.01   1772.9±4.75µs        ? ?/sec
   physical_plan_clickbench_q26                          1.00   1641.1±4.19µs        ? ?/sec    1.00   1648.7±4.56µs        ? ?/sec
   physical_plan_clickbench_q27                          1.01   1805.5±7.64µs        ? ?/sec    1.00   1789.2±5.74µs        ? ?/sec
   physical_plan_clickbench_q28                          1.00      2.1±0.01ms        ? ?/sec    1.00      2.1±0.00ms        ? ?/sec
   physical_plan_clickbench_q29                          1.02      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q3                           1.03   1583.9±8.48µs        ? ?/sec    1.00   1534.2±4.87µs        ? ?/sec
   physical_plan_clickbench_q30                          1.01     12.6±0.04ms        ? ?/sec    1.00     12.5±0.10ms        ? ?/sec
   physical_plan_clickbench_q31                          1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q32                          1.00      2.2±0.01ms        ? ?/sec    1.01      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q33                          1.00   1862.3±7.19µs        ? ?/sec    1.00   1855.5±6.40µs        ? ?/sec
   physical_plan_clickbench_q34                          1.00   1682.5±5.16µs        ? ?/sec    1.01   1705.1±6.57µs        ? ?/sec
   physical_plan_clickbench_q35                          1.00   1711.6±4.34µs        ? ?/sec    1.00   1704.2±6.07µs        ? ?/sec
   physical_plan_clickbench_q36                          1.00   1894.1±5.08µs        ? ?/sec    1.01  1911.2±11.57µs        ? ?/sec
   physical_plan_clickbench_q37                          1.00      2.4±0.01ms        ? ?/sec    1.02      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q38                          1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q39                          1.00      2.4±0.02ms        ? ?/sec    1.00      2.5±0.01ms        ? ?/sec
   physical_plan_clickbench_q4                           1.00   1350.0±9.02µs        ? ?/sec    1.00   1349.2±4.35µs        ? ?/sec
   physical_plan_clickbench_q40                          1.01      3.1±0.01ms        ? ?/sec    1.00      3.1±0.01ms        ? ?/sec
   physical_plan_clickbench_q41                          1.00      2.7±0.01ms        ? ?/sec    1.00      2.8±0.01ms        ? ?/sec
   physical_plan_clickbench_q42                          1.00      2.7±0.01ms        ? ?/sec    1.03      2.8±0.01ms        ? ?/sec
   physical_plan_clickbench_q43                          1.00      2.9±0.01ms        ? ?/sec    1.02      2.9±0.01ms        ? ?/sec
   physical_plan_clickbench_q44                          1.00   1448.9±4.96µs        ? ?/sec    1.00   1442.4±4.51µs        ? ?/sec
   physical_plan_clickbench_q45                          1.00   1432.3±5.29µs        ? ?/sec    1.01   1440.6±6.47µs        ? ?/sec
   physical_plan_clickbench_q46                          1.00   1694.1±4.32µs        ? ?/sec    1.00   1687.0±6.09µs        ? ?/sec
   physical_plan_clickbench_q47                          1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q48                          1.00      2.4±0.01ms        ? ?/sec    1.00      2.4±0.01ms        ? ?/sec
   physical_plan_clickbench_q49                          1.02      2.5±0.01ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
   physical_plan_clickbench_q5                           1.03  1521.1±31.67µs        ? ?/sec    1.00   1471.7±7.61µs        ? ?/sec
   physical_plan_clickbench_q50                          1.00      2.5±0.01ms        ? ?/sec    1.05      2.6±0.02ms        ? ?/sec
   physical_plan_clickbench_q51                          1.00   1833.0±5.36µs        ? ?/sec    1.00   1827.5±6.30µs        ? ?/sec
   physical_plan_clickbench_q52                          1.00      2.3±0.01ms        ? ?/sec    1.01      2.3±0.01ms        ? ?/sec
   physical_plan_clickbench_q53                          1.00   1653.4±4.95µs        ? ?/sec    1.00   1660.9±6.02µs        ? ?/sec
   physical_plan_clickbench_q54                          1.01   1653.8±5.20µs        ? ?/sec    1.00   1632.5±7.56µs        ? ?/sec
   physical_plan_clickbench_q55                          1.01   1592.4±5.54µs        ? ?/sec    1.00  1574.6±11.10µs        ? ?/sec
   physical_plan_clickbench_q56                          1.00   1573.9±4.77µs        ? ?/sec    1.00   1574.5±4.74µs        ? ?/sec
   physical_plan_clickbench_q57                          1.00   1722.7±9.72µs        ? ?/sec    1.00   1719.0±5.84µs        ? ?/sec
   physical_plan_clickbench_q58                          1.00      2.1±0.03ms        ? ?/sec    1.03      2.2±0.01ms        ? ?/sec
   physical_plan_clickbench_q59                          1.00   1619.6±5.37µs        ? ?/sec    1.00   1624.4±6.28µs        ? ?/sec
   physical_plan_clickbench_q6                           1.00   1462.5±4.12µs        ? ?/sec    1.02   1484.6±5.46µs        ? ?/sec
   physical_plan_clickbench_q60                          1.00   1600.7±3.51µs        ? ?/sec    1.01   1622.1±4.95µs        ? ?/sec
   physical_plan_clickbench_q7                           1.00   1332.7±4.63µs        ? ?/sec    1.01  1346.6±18.42µs        ? ?/sec
   physical_plan_clickbench_q8                           1.00   1807.8±5.85µs        ? ?/sec    1.05  1890.0±15.37µs        ? ?/sec
   physical_plan_clickbench_q9                           1.00   1735.5±4.93µs        ? ?/sec    1.01   1757.5±7.94µs        ? ?/sec
   physical_plan_struct_join_agg_sort                    1.00   1201.3±2.92µs        ? ?/sec    1.01  1211.2±11.59µs        ? ?/sec
   physical_plan_tpcds_all                               1.00    562.8±7.16ms        ? ?/sec    1.01    567.7±3.26ms        ? ?/sec
   physical_plan_tpch_all                                1.00     39.2±0.08ms        ? ?/sec    1.01     39.6±0.05ms        ? ?/sec
   physical_plan_tpch_q1                                 1.00   1373.7±9.91µs        ? ?/sec    1.02   1403.3±3.80µs        ? ?/sec
   physical_plan_tpch_q10                                1.00      2.0±0.00ms        ? ?/sec    1.02      2.1±0.00ms        ? ?/sec
   physical_plan_tpch_q11                                1.00      2.1±0.00ms        ? ?/sec    1.01      2.1±0.00ms        ? ?/sec
   physical_plan_tpch_q12                                1.00   1192.7±2.56µs        ? ?/sec    1.01   1210.5±2.57µs        ? ?/sec
   physical_plan_tpch_q13                                1.00    972.5±5.00µs        ? ?/sec    1.02    996.7±4.78µs        ? ?/sec
   physical_plan_tpch_q14                                1.00   1242.7±2.31µs        ? ?/sec    1.02   1269.5±2.56µs        ? ?/sec
   physical_plan_tpch_q16                                1.00  1507.4±11.92µs        ? ?/sec    1.01   1522.2±1.97µs        ? ?/sec
   physical_plan_tpch_q17                                1.00   1506.0±9.44µs        ? ?/sec    1.00   1503.4±4.05µs        ? ?/sec
   physical_plan_tpch_q18                                1.00   1731.2±3.33µs        ? ?/sec    1.02   1773.4±2.68µs        ? ?/sec
   physical_plan_tpch_q19                                1.00   1960.4±3.38µs        ? ?/sec    1.00   1964.8±3.51µs        ? ?/sec
   physical_plan_tpch_q2                                 1.00      3.2±0.00ms        ? ?/sec    1.02      3.3±0.02ms        ? ?/sec
   physical_plan_tpch_q20                                1.00   1937.7±3.10µs        ? ?/sec    1.02   1983.3±4.89µs        ? ?/sec
   physical_plan_tpch_q21                                1.00      2.4±0.00ms        ? ?/sec    1.01      2.4±0.00ms        ? ?/sec
   physical_plan_tpch_q22                                1.00   1452.8±2.72µs        ? ?/sec    1.04   1512.8±4.63µs        ? ?/sec
   physical_plan_tpch_q3                                 1.02  1675.2±80.24µs        ? ?/sec    1.00   1647.1±5.41µs        ? ?/sec
   physical_plan_tpch_q4                                 1.04   1096.6±7.83µs        ? ?/sec    1.00   1050.1±3.40µs        ? ?/sec
   physical_plan_tpch_q5                                 1.00      2.1±0.01ms        ? ?/sec    1.02      2.2±0.02ms        ? ?/sec
   physical_plan_tpch_q6                                 1.01    698.4±1.48µs        ? ?/sec    1.00    694.5±1.60µs        ? ?/sec
   physical_plan_tpch_q7                                 1.00      2.5±0.00ms        ? ?/sec    1.00      2.5±0.00ms        ? ?/sec
   physical_plan_tpch_q8                                 1.00      3.3±0.03ms        ? ?/sec    1.01      3.4±0.01ms        ? ?/sec
   physical_plan_tpch_q9                                 1.00      2.5±0.01ms        ? ?/sec    1.01      2.5±0.01ms        ? ?/sec
   physical_select_aggregates_from_200                   1.00      7.3±0.01ms        ? ?/sec    1.01      7.4±0.01ms        ? ?/sec
   physical_select_all_from_1000                         1.00     16.9±0.08ms        ? ?/sec    1.06     18.0±0.06ms        ? ?/sec
   physical_select_one_from_700                          1.00    643.6±2.38µs        ? ?/sec    1.00    641.2±3.39µs        ? ?/sec
   physical_sorted_union_order_by_10_int64               1.00      3.5±0.01ms        ? ?/sec    1.02      3.5±0.02ms        ? ?/sec
   physical_sorted_union_order_by_10_uint64              1.00      6.7±0.02ms        ? ?/sec    1.02      6.8±0.03ms        ? ?/sec
   physical_sorted_union_order_by_50_int64               1.00     75.2±0.19ms        ? ?/sec    1.00     75.4±0.27ms        ? ?/sec
   physical_sorted_union_order_by_50_uint64              1.00    254.2±2.08ms        ? ?/sec    1.01    256.1±1.89ms        ? ?/sec
   physical_theta_join_consider_sort                     1.00    981.1±3.05µs        ? ?/sec    1.00    977.7±1.66µs        ? ?/sec
   physical_unnest_to_join                               1.00    543.8±2.10µs        ? ?/sec    1.01    550.4±2.07µs        ? ?/sec
   physical_window_function_partition_by_12_on_values    1.01    626.6±1.19µs        ? ?/sec    1.00    619.8±1.54µs        ? ?/sec
   physical_window_function_partition_by_30_on_values    1.01   1211.3±4.31µs        ? ?/sec    1.00   1196.9±2.01µs        ? ?/sec
   physical_window_function_partition_by_4_on_values     1.04    400.9±1.06µs        ? ?/sec    1.00    385.1±0.91µs        ? ?/sec
   physical_window_function_partition_by_7_on_values     1.03    482.3±1.35µs        ? ?/sec    1.00    469.3±0.92µs        ? ?/sec
   physical_window_function_partition_by_8_on_values     1.00    506.8±0.89µs        ? ?/sec    1.00    508.5±1.26µs        ? ?/sec
   with_param_values_many_columns                        1.00    370.3±1.78µs        ? ?/sec    1.01    373.2±1.55µs        ? ?/sec
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **sql_planner — base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 1785.4s |
   | Peak memory | 140.3 MiB |
   | Avg memory | 103.3 MiB |
   | CPU user | 1979.9s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **sql_planner — branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 1955.5s |
   | Peak memory | 143.4 MiB |
   | Avg memory | 97.0 MiB |
   | CPU user | 1982.8s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6425374483

   After all the discussion on https://github.com/apache/datafusion/issues/26481#issuecomment-6416652913 is this one we should review / merge @jayzhan211  in your opinion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4390035358


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3596,6 +3608,69 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {
     }
 }
 
+/// A physical optimizer rule and the plan it produced.
+type PlanAfterRule<'a> = (
+    &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,
+    Arc<dyn ExecutionPlan>,
+);
+
+/// Adds context to an error from the final [`check_plan_tree_sanity`].
+///
+/// `EnsureRequirements` makes the order and distribution requirements hold,
+/// and every rule after it must keep them. When they do not hold for
+/// `final_plan`, the context names the rule that broke this contract: either
+/// `EnsureRequirements` itself, or the rule after which the requirements
+/// stopped holding.
+fn sanity_check_error(
+    error: DataFusionError,
+    final_plan: &Arc<dyn ExecutionPlan>,
+    input_plan: &Arc<dyn ExecutionPlan>,
+    plan_after_rule: &[PlanAfterRule],
+) -> DataFusionError {
+    const SANITY_CHECK: &str = "SanityCheckPlan";
+    if input_requirements_hold(final_plan) {
+        // The error is about unbounded inputs or another invariant.
+        return DataFusionError::Context(SANITY_CHECK.to_string(), Box::new(error));
+    }
+
+    let ensure_requirements = EnsureRequirements::new();
+    let enforced = plan_after_rule
+        .iter()
+        .rposition(|(rule, _)| rule.name() == ensure_requirements.name());

Review Comment:
   Once #26757 lands, this search could start at the enforcement phase boundary instead of matching `EnsureRequirements` by name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6441291493

   > After all the discussion on [#26481 (comment)](https://github.com/apache/datafusion/issues/26481#issuecomment-6416652913) is this one we should review / merge @jayzhan211 in your opinion?
   
   We can review/merge this, they could be parallel


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4401991298


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3596,6 +3608,69 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {
     }
 }
 
+/// A physical optimizer rule and the plan it produced.
+type PlanAfterRule<'a> = (
+    &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,
+    Arc<dyn ExecutionPlan>,
+);
+
+/// Adds context to an error from the final [`check_plan_tree_sanity`].
+///
+/// `EnsureRequirements` makes the order and distribution requirements hold,
+/// and every rule after it must keep them. When they do not hold for
+/// `final_plan`, the context names the rule that broke this contract: either
+/// `EnsureRequirements` itself, or the rule after which the requirements
+/// stopped holding.
+fn sanity_check_error(
+    error: DataFusionError,
+    final_plan: &Arc<dyn ExecutionPlan>,
+    input_plan: &Arc<dyn ExecutionPlan>,
+    plan_after_rule: &[PlanAfterRule],
+) -> DataFusionError {
+    const SANITY_CHECK: &str = "SanityCheckPlan";
+    if input_requirements_hold(final_plan) {
+        // The error is about unbounded inputs or another invariant.
+        return DataFusionError::Context(SANITY_CHECK.to_string(), Box::new(error));
+    }
+
+    let ensure_requirements = EnsureRequirements::new();
+    let enforced = plan_after_rule
+        .iter()
+        .rposition(|(rule, _)| rule.name() == ensure_requirements.name());

Review Comment:
   #26757 is still a draft, so I'll make that switch there: once it is rebased on this PR, the search will start at the analyzer/optimizer boundary and the name match goes away.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4417178213


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3391,6 +3395,10 @@ impl DefaultPhysicalPlanner {
 
         let mut new_plan = Arc::clone(&plan);
         let optimizer_context = SessionOptimizerContext::new(session_state);
+        // The plan after each rule, so that a failed final check can name the
+        // rule that broke the plan. Plans share unchanged subtrees, so this
+        // costs one pointer per rule.

Review Comment:
   it also pins the plan memory in place for longer, so it would make peak memory usage higher which maybe doesn't matter, but it is a differene



##########
datafusion/core/tests/physical_optimizer/sanity_checker.rs:
##########
@@ -999,3 +1003,116 @@ async fn test_union_with_sorts_and_constants() -> Result<()> {
 
     Ok(())
 }
+
+/// Removes every [`SortExec`], standing in for a rule that breaks ordering

Review Comment:
   FWIW I was confused at first why this is needed. It is a test it turns out. Maybe calling it `MockRule` or adding a comment like '/// Test rule to check messages' would be more clear



##########
docs/source/library-user-guide/upgrading/56.0.0.md:
##########
@@ -1336,3 +1336,34 @@ without declaring one, use `FileScanConfigBuilder::with_preserve_order(true)`.
 Code that reads `FileScanConfig::preserve_order` to decide whether files may be
 reordered should call `FileScanConfig::preserves_file_order()` instead, which
 also accounts for the output ordering.
+
+### `SanityCheckPlan` is deprecated and no longer in the default physical optimizer rules
+
+The physical planner now checks the final physical plan itself, after the last

Review Comment:
   I think we could probably shorten this and note that the `SanityCheckPlan` has been replaced with a method and if custom pipelines need to check the plan they should call the new method directly



##########
datafusion/core/tests/physical_optimizer/sanity_checker.rs:
##########
@@ -999,3 +1003,116 @@ async fn test_union_with_sorts_and_constants() -> Result<()> {
 
     Ok(())
 }
+
+/// Removes every [`SortExec`], standing in for a rule that breaks ordering
+/// requirements.
+///
+/// [`SortExec`]: datafusion_physical_plan::sorts::sort::SortExec
+#[derive(Debug)]
+struct RemoveSorts;
+
+impl PhysicalOptimizerRule for RemoveSorts {
+    fn optimize(
+        &self,
+        plan: Arc<dyn ExecutionPlan>,
+        _config: &ConfigOptions,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        plan.transform_up(|plan| {
+            if plan.is::<SortExec>() {
+                Ok(Transformed::yes(Arc::clone(plan.children()[0])))
+            } else {
+                Ok(Transformed::no(plan))
+            }
+        })
+        .data()
+    }
+
+    fn name(&self) -> &str {
+        "remove_sorts"
+    }
+
+    fn schema_check(&self) -> bool {
+        true
+    }
+}
+
+async fn plan_window_query(state: SessionState) -> Result<Arc<dyn ExecutionPlan>> {
+    let ctx = SessionContext::new_with_state(state);
+    ctx.sql("CREATE TABLE t(a INT) AS VALUES (3), (1), (2)")
+        .await?
+        .collect()
+        .await?;
+    ctx.sql("SELECT a, row_number() OVER (ORDER BY a) AS rn FROM t")
+        .await?
+        .create_physical_plan()
+        .await
+}
+
+/// A rule appended after the default rules is checked, and the error names it.
+#[tokio::test]
+async fn test_final_check_names_appended_rule() -> Result<()> {
+    let state = SessionStateBuilder::new()
+        .with_default_features()
+        .with_physical_optimizer_rule(Arc::new(RemoveSorts))
+        .build();
+    let error = plan_window_query(state).await.unwrap_err().to_string();
+    assert!(
+        error.starts_with(
+            "SanityCheckPlan: order and distribution requirements stopped holding \
+             after physical optimizer rule 'remove_sorts'"
+        ),
+        "{error}"
+    );
+    assert!(
+        error.contains("does not satisfy order requirements"),
+        "{error}"
+    );
+    Ok(())
+}
+
+/// A custom rule list is checked too.
+#[tokio::test]
+async fn test_final_check_names_rule_in_custom_pipeline() -> Result<()> {

Review Comment:
   How is this different than test_final_check_names_appended_rule ?



##########
datafusion/core/tests/memory_limit/mod.rs:
##########
@@ -578,7 +578,7 @@ async fn sort_merge_join_no_spill() {
         ])
         .with_memory_limit(1_000)
         .with_config(config)
-        .with_scenario(Scenario::AccessLogStreaming)
+        .with_scenario(Scenario::AccessLogStreamingByPodTime)

Review Comment:
   This change seems unrelated to the rest of this PR? Or on more review it seems related to forcing an error. However it is really strange to me that we are reusing the tests for memory limitiing to test optimizer errors. I would suggest we move the optimizer error stuff into an optimizer_integration style test



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3344,12 +3352,16 @@ impl DefaultPhysicalPlanner {
                 optimizer.name(),
                 displayable(new_plan.as_ref()).indent(false)
             );
-            observer(new_plan.as_ref(), optimizer.as_ref())
+            observer(new_plan.as_ref(), optimizer.as_ref());
+            plan_after_rule.push((optimizer, Arc::clone(&new_plan)));
         }
 
-        // This runs once after all optimizer runs are complete,
-        // to verify that the plan is executable.
-        InvariantChecker(InvariantLevel::Executable).check(&new_plan)?;
+        // This runs once after all optimizer runs are complete, including
+        // rules appended by the user, to verify that the plan is executable.
+        let optimizer_options = &session_state.config_options().optimizer;
+        if let Err(e) = check_plan_tree_sanity(&new_plan, optimizer_options) {

Review Comment:
   I actually think we shouldn't allow plans to opt out of plan sanity checking -- if the generated plan doesn't pass sanity check then it will fail in some other way (either wrong results or worse). Catching it early is better I think



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3391,6 +3395,10 @@ impl DefaultPhysicalPlanner {
 
         let mut new_plan = Arc::clone(&plan);
         let optimizer_context = SessionOptimizerContext::new(session_state);
+        // The plan after each rule, so that a failed final check can name the
+        // rule that broke the plan. Plans share unchanged subtrees, so this
+        // costs one pointer per rule.

Review Comment:
   Also there are a bunch of rules that try to `unwrap` the Arc to avoid allocations which this will prevent.
   
   If you only want to get the plans on failure, I recommend saving only the input plan and if there is a failure then go back and run each of the passes (again) to get a more specific message



##########
datafusion/sqllogictest/test_files/explain.slt:
##########
@@ -242,7 +242,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
 physical_plan after PushdownSort SAME TEXT AS ABOVE
 physical_plan after EnsureCooperative SAME TEXT AS ABOVE
 physical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE
-physical_plan after SanityCheckPlan SAME TEXT AS ABOVE

Review Comment:
   I do like that this is no longer an OptimizerRule



##########
datafusion/core/tests/physical_optimizer/sanity_checker.rs:
##########
@@ -999,3 +1003,116 @@ async fn test_union_with_sorts_and_constants() -> Result<()> {
 
     Ok(())
 }
+
+/// Removes every [`SortExec`], standing in for a rule that breaks ordering
+/// requirements.
+///
+/// [`SortExec`]: datafusion_physical_plan::sorts::sort::SortExec
+#[derive(Debug)]
+struct RemoveSorts;
+
+impl PhysicalOptimizerRule for RemoveSorts {
+    fn optimize(
+        &self,
+        plan: Arc<dyn ExecutionPlan>,
+        _config: &ConfigOptions,
+    ) -> Result<Arc<dyn ExecutionPlan>> {
+        plan.transform_up(|plan| {
+            if plan.is::<SortExec>() {
+                Ok(Transformed::yes(Arc::clone(plan.children()[0])))
+            } else {
+                Ok(Transformed::no(plan))
+            }
+        })
+        .data()
+    }
+
+    fn name(&self) -> &str {
+        "remove_sorts"
+    }
+
+    fn schema_check(&self) -> bool {
+        true
+    }
+}
+
+async fn plan_window_query(state: SessionState) -> Result<Arc<dyn ExecutionPlan>> {
+    let ctx = SessionContext::new_with_state(state);
+    ctx.sql("CREATE TABLE t(a INT) AS VALUES (3), (1), (2)")
+        .await?
+        .collect()
+        .await?;
+    ctx.sql("SELECT a, row_number() OVER (ORDER BY a) AS rn FROM t")
+        .await?
+        .create_physical_plan()
+        .await
+}
+
+/// A rule appended after the default rules is checked, and the error names it.
+#[tokio::test]
+async fn test_final_check_names_appended_rule() -> Result<()> {
+    let state = SessionStateBuilder::new()
+        .with_default_features()
+        .with_physical_optimizer_rule(Arc::new(RemoveSorts))
+        .build();
+    let error = plan_window_query(state).await.unwrap_err().to_string();
+    assert!(
+        error.starts_with(
+            "SanityCheckPlan: order and distribution requirements stopped holding \
+             after physical optimizer rule 'remove_sorts'"
+        ),
+        "{error}"
+    );
+    assert!(
+        error.contains("does not satisfy order requirements"),
+        "{error}"
+    );
+    Ok(())
+}
+
+/// A custom rule list is checked too.
+#[tokio::test]
+async fn test_final_check_names_rule_in_custom_pipeline() -> Result<()> {
+    let state = SessionStateBuilder::new()
+        .with_default_features()
+        .with_physical_optimizer_rules(vec![
+            Arc::new(EnsureRequirements::new()),
+            Arc::new(RemoveSorts),
+        ])
+        .build();
+    let error = plan_window_query(state).await.unwrap_err().to_string();
+    assert!(
+        error.starts_with(
+            "SanityCheckPlan: order and distribution requirements stopped holding \
+             after physical optimizer rule 'remove_sorts'"
+        ),
+        "{error}"
+    );
+    Ok(())
+}
+
+/// Without `EnsureRequirements`, the window's sort is never added.
+#[tokio::test]
+async fn test_final_check_without_ensure_requirements() -> Result<()> {
+    let state = SessionStateBuilder::new()
+        .with_default_features()
+        .with_physical_optimizer_rules(vec![Arc::new(RemoveSorts)])
+        .build();
+    let error = plan_window_query(state).await.unwrap_err().to_string();
+    assert!(
+        error.starts_with(
+            "SanityCheckPlan: no physical optimizer rule produced a plan that \
+             satisfies the order and distribution requirements"
+        ),
+        "{error}"
+    );
+    Ok(())
+}
+
+/// The default rules produce a plan that passes the final check.
+#[tokio::test]
+async fn test_final_check_passes_default_rules() -> Result<()> {

Review Comment:
   this I think is redundant as it is tested many times in the rest of the code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#discussion_r4307184199


##########
datafusion/core/src/optimizer_rule_reference.md:
##########
@@ -69,26 +69,30 @@ Rule order matters. The default pipeline may change between releases.
 The same rule name may appear more than once when the default pipeline runs it
 in multiple phases.
 
-| order | rule                           | phase                   | summary                                                                                                      |
-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
-| 1     | `OutputRequirements`           | add phase               | Adds helper nodes so output requirements survive later physical rewrites.                                    |
-| 2     | `aggregate_statistics`         | -                       | Uses exact source statistics to answer some aggregates without scanning data.                                |
-| 3     | `join_selection`               | -                       | Chooses join implementation, build side, and partition mode from statistics and stream properties.           |
-| 4     | `LimitedDistinctAggregation`   | -                       | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed.              |
-| 5     | `FilterPushdown`               | pre-optimization phase  | Pushes supported physical filters down toward data sources before distribution and sorting are enforced.     |
-| 6     | `WindowTopN`                   | -                       | Replaces eligible row-number window and filter patterns with per-partition TopK execution.                   |
-| 7     | `EnsureRequirements`           | -                       | Enforces both distribution and sorting requirements in a single idempotent rule.                             |
-| 8     | `CombinePartialFinalAggregate` | -                       | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.             |
-| 9     | `OptimizeAggregateOrder`       | -                       | Updates aggregate expressions to use the best ordering once sort requirements are known.                     |
-| 10    | `ProjectionPushdown`           | early pass              | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure.           |
-| 11    | `OutputRequirements`           | remove phase            | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done.          |
-| 12    | `LimitAggregation`             | -                       | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets.                   |
-| 13    | `LimitPushPastWindows`         | -                       | Pushes fetch limits through bounded window operators when doing so keeps the result correct.                 |
-| 14    | `HashJoinBuffering`            | -                       | Adds buffering on the probe side of hash joins so probing can start before build completion.                 |
-| 15    | `LimitPushdown`                | -                       | Moves physical limits into child operators or fetch-enabled variants to cut data early.                      |
-| 16    | `TopKRepartition`              | -                       | Pushes TopK below hash repartition when the partition key is a prefix of the sort key.                       |
-| 17    | `ProjectionPushdown`           | late pass               | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities.               |
-| 18    | `PushdownSort`                 | -                       | Pushes sort requirements into data sources that can already return sorted output.                            |
-| 19    | `EnsureCooperative`            | -                       | Wraps non-cooperative plan parts so long-running tasks yield fairly.                                         |
-| 20    | `FilterPushdown(Post)`         | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving.                        |
-| 21    | `SanityCheckPlan`              | -                       | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
+After the last rule, including any rule added by the user, the physical planner

Review Comment:
   TIL about this document 🤯 



##########
datafusion/physical-optimizer/src/optimizer.rs:
##########
@@ -196,15 +195,10 @@ impl PhysicalOptimizer {
             // Therefore, it should be run at the end of the optimization process since any changes to the plan may break the dynamic filter's references.
             // See `FilterPushdownPhase` for more details.
             Arc::new(FilterPushdown::new_post_optimization()),
-            // The SanityCheckPlan rule checks whether the order and
-            // distribution requirements of each node in the plan
-            // is satisfied. It will also reject non-runnable query
-            // plans that use pipeline-breaking operators on infinite
-            // input(s). The rule generates a diagnostic error
-            // message for invalid plans. It makes no changes to the
-            // given query plan; i.e. it only acts as a final
-            // gatekeeping rule.
-            Arc::new(SanityCheckPlan::new()),
+            // There is no `SanityCheckPlan` rule at the end of this list. The

Review Comment:
   how about we deprecate `SanityCheckplan` rule so it is clearer we intend to remove it



##########
datafusion/core/src/optimizer_rule_reference.md:
##########
@@ -69,26 +69,30 @@ Rule order matters. The default pipeline may change between releases.
 The same rule name may appear more than once when the default pipeline runs it
 in multiple phases.
 
-| order | rule                           | phase                   | summary                                                                                                      |
-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
-| 1     | `OutputRequirements`           | add phase               | Adds helper nodes so output requirements survive later physical rewrites.                                    |
-| 2     | `aggregate_statistics`         | -                       | Uses exact source statistics to answer some aggregates without scanning data.                                |
-| 3     | `join_selection`               | -                       | Chooses join implementation, build side, and partition mode from statistics and stream properties.           |
-| 4     | `LimitedDistinctAggregation`   | -                       | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed.              |
-| 5     | `FilterPushdown`               | pre-optimization phase  | Pushes supported physical filters down toward data sources before distribution and sorting are enforced.     |
-| 6     | `WindowTopN`                   | -                       | Replaces eligible row-number window and filter patterns with per-partition TopK execution.                   |
-| 7     | `EnsureRequirements`           | -                       | Enforces both distribution and sorting requirements in a single idempotent rule.                             |
-| 8     | `CombinePartialFinalAggregate` | -                       | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.             |
-| 9     | `OptimizeAggregateOrder`       | -                       | Updates aggregate expressions to use the best ordering once sort requirements are known.                     |
-| 10    | `ProjectionPushdown`           | early pass              | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure.           |
-| 11    | `OutputRequirements`           | remove phase            | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done.          |
-| 12    | `LimitAggregation`             | -                       | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets.                   |
-| 13    | `LimitPushPastWindows`         | -                       | Pushes fetch limits through bounded window operators when doing so keeps the result correct.                 |
-| 14    | `HashJoinBuffering`            | -                       | Adds buffering on the probe side of hash joins so probing can start before build completion.                 |
-| 15    | `LimitPushdown`                | -                       | Moves physical limits into child operators or fetch-enabled variants to cut data early.                      |
-| 16    | `TopKRepartition`              | -                       | Pushes TopK below hash repartition when the partition key is a prefix of the sort key.                       |
-| 17    | `ProjectionPushdown`           | late pass               | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities.               |
-| 18    | `PushdownSort`                 | -                       | Pushes sort requirements into data sources that can already return sorted output.                            |
-| 19    | `EnsureCooperative`            | -                       | Wraps non-cooperative plan parts so long-running tasks yield fairly.                                         |
-| 20    | `FilterPushdown(Post)`         | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving.                        |
-| 21    | `SanityCheckPlan`              | -                       | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
+After the last rule, including any rule added by the user, the physical planner

Review Comment:
   I feel like this document is outdated and redundand and hard to find -- maybe as a follow on PR we can fold it into the documentation in https://github.com/apache/datafusion/pull/26351
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6250468061

   🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26461#issuecomment-6250437807)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6250437807-3566-6xl96 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing planner-final-plan-check (2689cbbfae6c63ce9073b0a95cf09c4328fc75d7) to 290ad15 (merge-base) [diff](https://github.com/apache/datafusion/compare/290ad15cc842167ca4d3c0774ccdf3a1d9d61862..2689cbbfae6c63ce9073b0a95cf09c4328fc75d7)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark sql_planner
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Check the final physical plan in the planner and name the rule that broke it [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26461:
URL: https://github.com/apache/datafusion/pull/26461#issuecomment-6268896852

   @2010YOUY01 Thanks. To see who would need that API, I looked at how downstream projects use the rule today:
   
   - greptimedb moves the rule to the end of its list. sail and cube add it to their own rule lists. The planner check now covers all three, and the deprecation warning tells them they can drop the rule.
   - spiceai looks up the rule's position in the default list and panics if the rule is missing. The upgrade guide covers that case.
   - openobserve places the rule before its own rules (`JoinReorderRule`, `RemoteScanRule` and others), with a comment saying those rules are not checked. streamling replaces the default rules and leaves out both `EnforceDistribution` and `SanityCheckPlan`. After this PR, both projects' plans are checked.
   
   The last group is the one your comment is about. If the check rejects a correct plan, for example one whose ordering holds but cannot be proven from the plan's properties, those projects can no longer skip the check. I'd suggest adding a way to skip or replace the check in a follow-up, once someone needs it, rather than guessing at the shape of the API now. I'm happy to add it here instead if you'd prefer it to ship in the same release.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org