You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/10/16 19:46:21 UTC
[PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
adriangb opened a new pull request, #26428:
URL: https://github.com/apache/datafusion/pull/26428
## Which issue does this PR close?
- Closes apache/datafusion#26424.
Stacked on apache/datafusion#26271. Only the last commit is new.
## Rationale for this change
A reference to a `MATERIALIZED` CTE reports no ordering and unknown partitioning. The continuation then sorts or repartitions rows that the body already sorted or partitioned:
```sql
WITH c AS MATERIALIZED (SELECT g, count(*) AS n FROM p GROUP BY g)
SELECT g, sum(n) FROM c GROUP BY g;
```
Before:
```
AggregateExec: mode=FinalPartitioned, gby=[g@0 as g], aggr=[sum(c.n)]
RepartitionExec: partitioning=Hash([g@0], 12), input_partitions=12
AggregateExec: mode=Partial, gby=[g@0 as g], aggr=[sum(c.n)]
MaterializedCteScanExec: name=c
```
After:
```
AggregateExec: mode=SinglePartitioned, gby=[g@0 as g], aggr=[sum(c.n)]
MaterializedCteScanExec: name=c, partitioning=Hash([g@0], 12)
```
On 10 million rows with 1 million groups (12 partitions, `datafusion-cli`, release build, median of 30 runs), the query takes about 35 ms instead of about 41 ms.
## What changes are included in this PR?
`DefaultPhysicalPlanner::optimize_physical_plan` now does these steps for a plan with materialized CTEs:
1. It optimizes each body on its own, as the child of a `MaterializedCteExec` with an empty continuation, so the body has the same parent as in the full plan. A body that reads another CTE goes after that CTE.
2. It gives each scan the partition count, the output ordering and the hash partitioning of its optimized body.
3. It optimizes the full plan one time.
4. It keeps the result only if each body in it still has the properties that its scans report. If not, it optimizes and uses the plan with statistics only.
There is no retry loop. The cost is one full optimizer pass plus one pass for each body, and one more full pass only when step 4 fails.
A scan with the properties of the body replays one body partition per partition, and returns an internal error if the partition counts are not the same at execution. A scan that comes from a separate planning run, such as a view, is bound again, so that it loses properties from that run.
Planning time per query, physical and logical (`EXPLAIN`, 2000 runs, median of 9):
| Query | Statistics only (#26271) | Repeated full passes (first version of #26271) | This PR |
| --- | --- | --- | --- |
| One CTE | 0.36 ms | 0.88 ms | 0.40 ms |
| Chain of 6 CTEs | 1.43 ms | 1.93 ms | 1.60 ms |
Limits:
- An `ORDER BY` at the top of a body without `LIMIT` is removed by `EnforceSorting`, as before, so the scan has no ordering in that case.
- `EXPLAIN VERBOSE` shows the rules of the full optimizer run that made the final plan, not the runs on the bodies.
## Are these changes tested?
Yes. `cte_materialized.slt`: a sort that is removed, a sort that is still needed, a body `ORDER BY` without `LIMIT`, a repartition and a partial aggregate that are removed, a partitioned join of two references, a chain of two CTEs, a CTE defined inside a CTE body, a scalar subquery and a view. Unit tests for the copied properties, the replay order and the partition-count error. Planner tests check that the optimizer runs one time on the body and one time on the full plan, and that an optimizer rule that changes the body after the scans got its properties makes the planner use the plan with statistics only. A view test checks a view that is not inlined.
## Are there any user-facing changes?
Plans of queries with `MATERIALIZED` CTEs can change. `MaterializedCteScanExec` shows `partitioning=` and `output_ordering=` in EXPLAIN when it has them. New public functions: `MaterializedCteScanExec::with_body_properties`, `body_properties_hold`, `mirrors_body`, and `MaterializedCteExec::register`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6570899079
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26428#issuecomment-6569850863)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing mcte-body-first-properties (4824866f4a324b75674127d1646070fea4a71d0c) to 985bbbf (merge-base) [diff](https://github.com/apache/datafusion/compare/985bbbf60e14644d8945dce9342f6dd20a428f1a..4824866f4a324b75674127d1646070fea4a71d0c)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD mcte-body-first-properties
----- ---- --------------------------
logical_aggregate_with_join 1.00 343.4±1.71µs ? ?/sec 1.01 345.6±2.06µs ? ?/sec
logical_correlated_subquery_exists 1.00 205.1±0.79µs ? ?/sec 1.02 208.4±1.47µs ? ?/sec
logical_correlated_subquery_in 1.01 209.6±1.48µs ? ?/sec 1.00 208.3±2.04µs ? ?/sec
logical_distinct_many_columns 1.00 291.6±1.30µs ? ?/sec 1.00 292.6±1.16µs ? ?/sec
logical_join_4_with_agg_and_filter 1.01 198.0±0.83µs ? ?/sec 1.00 195.3±0.99µs ? ?/sec
logical_join_8_with_agg_sort_limit 1.01 344.2±1.90µs ? ?/sec 1.00 341.1±1.97µs ? ?/sec
logical_join_chain_16 1.00 611.3±1.82µs ? ?/sec 1.01 614.9±2.61µs ? ?/sec
logical_join_chain_4 1.02 75.3±0.31µs ? ?/sec 1.00 73.5±0.56µs ? ?/sec
logical_join_chain_8 1.00 197.6±0.78µs ? ?/sec 1.00 197.9±0.63µs ? ?/sec
logical_multiple_subqueries 1.00 391.5±1.27µs ? ?/sec 1.00 390.7±1.80µs ? ?/sec
logical_nested_cte_4_levels 1.00 199.9±0.81µs ? ?/sec 1.00 199.8±0.67µs ? ?/sec
logical_plan_struct_join_agg_sort 1.00 125.6±0.78µs ? ?/sec 1.00 125.2±0.87µs ? ?/sec
logical_plan_tpcds_all 1.00 73.6±0.28ms ? ?/sec 1.00 73.7±0.29ms ? ?/sec
logical_plan_tpch_all 1.00 5.3±0.04ms ? ?/sec 1.02 5.4±0.02ms ? ?/sec
logical_scalar_subquery 1.01 217.6±0.99µs ? ?/sec 1.00 215.6±0.93µs ? ?/sec
logical_select_all_from_1000 1.00 6.8±0.03ms ? ?/sec 1.04 7.1±0.02ms ? ?/sec
logical_select_one_from_700 1.00 227.0±1.75µs ? ?/sec 1.00 227.1±1.83µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.00 205.8±1.41µs ? ?/sec 1.01 208.5±1.51µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 195.1±1.34µs ? ?/sec 1.00 195.9±1.27µs ? ?/sec
logical_union_4_branches 1.00 306.4±2.40µs ? ?/sec 1.00 307.4±1.58µs ? ?/sec
logical_union_8_branches 1.00 619.3±2.86µs ? ?/sec 1.01 625.6±6.82µs ? ?/sec
logical_wide_aggregate_1000_exprs 1.00 38.5±0.11ms ? ?/sec 1.01 38.9±0.13ms ? ?/sec
logical_wide_aggregate_100_exprs 1.00 1562.4±4.94µs ? ?/sec 1.01 1577.0±5.77µs ? ?/sec
logical_wide_case_50_exprs 1.00 1709.3±5.81µs ? ?/sec 1.01 1731.2±4.17µs ? ?/sec
logical_wide_filter_200_predicates 1.00 1292.7±5.13µs ? ?/sec 1.01 1310.5±5.90µs ? ?/sec
logical_wide_filter_50_predicates 1.00 340.9±1.31µs ? ?/sec 1.01 345.4±1.24µs ? ?/sec
optimizer_correlated_exists 1.00 214.4±1.38µs ? ?/sec 1.01 217.4±0.60µs ? ?/sec
optimizer_join_4_with_agg_filter 1.00 393.0±1.32µs ? ?/sec 1.01 396.5±1.47µs ? ?/sec
optimizer_join_chain_4 1.00 153.0±0.36µs ? ?/sec 1.01 153.8±0.50µs ? ?/sec
optimizer_join_chain_8 1.00 528.0±1.22µs ? ?/sec 1.00 530.2±1.31µs ? ?/sec
optimizer_select_all_from_1000 1.01 6.8±0.02ms ? ?/sec 1.00 6.7±0.01ms ? ?/sec
optimizer_select_one_from_700 1.00 236.8±0.61µs ? ?/sec 1.01 238.0±1.56µs ? ?/sec
optimizer_tpcds_all 1.00 249.3±0.85ms ? ?/sec 1.00 250.5±2.13ms ? ?/sec
optimizer_tpch_all 1.00 15.1±0.05ms ? ?/sec 1.00 15.0±0.06ms ? ?/sec
optimizer_wide_aggregate_100 1.00 1739.4±11.51µs ? ?/sec 1.00 1745.1±2.74µs ? ?/sec
optimizer_wide_filter_200 1.00 4.0±0.01ms ? ?/sec 1.01 4.0±0.01ms ? ?/sec
physical_cte_materialized_chain_6 1.00 2.8±0.01ms ? ?/sec
physical_cte_materialized_one_ref 1.00 758.5±2.23µs ? ?/sec
physical_cte_materialized_two_refs 1.00 1024.8±2.54µs ? ?/sec
physical_cte_mixed 1.00 3.2±0.01ms ? ?/sec
physical_cte_not_materialized_chain_6 1.00 17.9±0.05ms ? ?/sec
physical_cte_not_materialized_one_ref 1.00 689.8±1.75µs ? ?/sec
physical_cte_not_materialized_two_refs 1.00 1203.1±3.35µs ? ?/sec
physical_cte_unmarked_chain_6 1.00 17.9±0.12ms ? ?/sec
physical_cte_unmarked_one_ref 1.00 667.5±2.12µs ? ?/sec
physical_cte_unmarked_two_refs 1.00 1189.0±2.69µs ? ?/sec
physical_intersection 1.00 501.1±2.12µs ? ?/sec 1.04 519.1±2.23µs ? ?/sec
physical_join_consider_sort 1.00 940.9±2.66µs ? ?/sec 1.03 965.1±1.73µs ? ?/sec
physical_join_distinct 1.00 187.0±1.45µs ? ?/sec 1.00 187.8±1.41µs ? ?/sec
physical_many_self_joins 1.00 7.3±0.05ms ? ?/sec 1.00 7.3±0.01ms ? ?/sec
physical_plan_clickbench_all 1.02 134.0±1.27ms ? ?/sec 1.00 131.7±0.59ms ? ?/sec
physical_plan_clickbench_q1 1.00 1288.2±7.21µs ? ?/sec 1.00 1291.5±7.98µs ? ?/sec
physical_plan_clickbench_q10 1.02 1882.6±4.98µs ? ?/sec 1.00 1839.8±13.12µs ? ?/sec
physical_plan_clickbench_q11 1.01 1997.4±6.08µs ? ?/sec 1.00 1975.7±8.19µs ? ?/sec
physical_plan_clickbench_q12 1.00 2.0±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q13 1.00 1896.7±5.99µs ? ?/sec 1.01 1912.4±15.33µs ? ?/sec
physical_plan_clickbench_q14 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_clickbench_q15 1.00 1953.7±6.25µs ? ?/sec 1.00 1955.2±11.57µs ? ?/sec
physical_plan_clickbench_q16 1.00 1630.9±3.52µs ? ?/sec 1.05 1708.1±10.82µs ? ?/sec
physical_plan_clickbench_q17 1.00 1688.6±4.43µs ? ?/sec 1.00 1693.7±5.86µs ? ?/sec
physical_plan_clickbench_q18 1.00 1514.5±5.47µs ? ?/sec 1.03 1566.4±13.46µs ? ?/sec
physical_plan_clickbench_q19 1.00 1843.3±5.60µs ? ?/sec 1.06 1949.2±17.30µs ? ?/sec
physical_plan_clickbench_q2 1.00 1652.4±6.61µs ? ?/sec 1.00 1645.5±8.24µs ? ?/sec
physical_plan_clickbench_q20 1.00 1467.5±5.42µs ? ?/sec 1.01 1481.4±9.34µs ? ?/sec
physical_plan_clickbench_q21 1.00 1652.4±4.31µs ? ?/sec 1.04 1715.1±23.79µs ? ?/sec
physical_plan_clickbench_q22 1.00 2.0±0.01ms ? ?/sec 1.02 2.0±0.02ms ? ?/sec
physical_plan_clickbench_q23 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.8±0.01ms ? ?/sec 1.03 4.9±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.00 1742.1±6.10µs ? ?/sec 1.04 1817.7±7.02µs ? ?/sec
physical_plan_clickbench_q26 1.00 1606.9±6.67µs ? ?/sec 1.02 1642.2±5.32µs ? ?/sec
physical_plan_clickbench_q27 1.00 1774.0±6.14µs ? ?/sec 1.02 1814.7±8.89µs ? ?/sec
physical_plan_clickbench_q28 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q29 1.00 2.2±0.01ms ? ?/sec 1.02 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q3 1.01 1540.3±6.23µs ? ?/sec 1.00 1531.6±4.85µs ? ?/sec
physical_plan_clickbench_q30 1.00 12.6±0.11ms ? ?/sec 1.01 12.7±0.07ms ? ?/sec
physical_plan_clickbench_q31 1.00 2.2±0.01ms ? ?/sec 1.02 2.2±0.01ms ? ?/sec
physical_plan_clickbench_q32 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
physical_plan_clickbench_q33 1.00 1849.7±8.06µs ? ?/sec 1.02 1886.2±8.06µs ? ?/sec
physical_plan_clickbench_q34 1.00 1664.1±6.55µs ? ?/sec 1.03 1707.5±11.51µs ? ?/sec
physical_plan_clickbench_q35 1.00 1690.5±4.37µs ? ?/sec 1.01 1712.6±6.17µs ? ?/sec
physical_plan_clickbench_q36 1.00 1918.8±5.19µs ? ?/sec 1.01 1945.9±10.09µs ? ?/sec
physical_plan_clickbench_q37 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q38 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q39 1.00 2.4±0.01ms ? ?/sec 1.01 2.4±0.02ms ? ?/sec
physical_plan_clickbench_q4 1.00 1384.3±12.24µs ? ?/sec 1.01 1396.0±5.73µs ? ?/sec
physical_plan_clickbench_q40 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q41 1.01 2.8±0.03ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q42 1.03 2.8±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_clickbench_q43 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
physical_plan_clickbench_q44 1.00 1473.2±6.71µs ? ?/sec 1.00 1467.5±7.63µs ? ?/sec
physical_plan_clickbench_q45 1.00 1472.0±5.70µs ? ?/sec 1.00 1473.8±5.72µs ? ?/sec
physical_plan_clickbench_q46 1.00 1734.9±10.74µs ? ?/sec 1.00 1742.8±7.66µs ? ?/sec
physical_plan_clickbench_q47 1.00 2.2±0.01ms ? ?/sec 1.01 2.3±0.02ms ? ?/sec
physical_plan_clickbench_q48 1.02 2.5±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_clickbench_q49 1.03 2.5±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_clickbench_q5 1.02 1504.8±14.08µs ? ?/sec 1.00 1479.7±9.28µs ? ?/sec
physical_plan_clickbench_q50 1.03 2.6±0.03ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_clickbench_q51 1.00 1801.3±10.75µs ? ?/sec 1.00 1803.7±7.25µs ? ?/sec
physical_plan_clickbench_q52 1.01 2.3±0.02ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
physical_plan_clickbench_q53 1.00 1695.3±10.68µs ? ?/sec 1.00 1687.8±7.25µs ? ?/sec
physical_plan_clickbench_q54 1.00 1687.6±14.54µs ? ?/sec 1.01 1711.7±7.75µs ? ?/sec
physical_plan_clickbench_q55 1.04 1684.3±24.62µs ? ?/sec 1.00 1616.2±7.76µs ? ?/sec
physical_plan_clickbench_q56 1.02 1638.9±23.48µs ? ?/sec 1.00 1599.2±5.24µs ? ?/sec
physical_plan_clickbench_q57 1.00 1736.6±20.52µs ? ?/sec 1.00 1740.3±17.03µs ? ?/sec
physical_plan_clickbench_q58 1.04 2.2±0.05ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
physical_plan_clickbench_q59 1.04 1687.5±29.61µs ? ?/sec 1.00 1628.6±8.42µs ? ?/sec
physical_plan_clickbench_q6 1.00 1478.6±6.81µs ? ?/sec 1.03 1517.6±8.74µs ? ?/sec
physical_plan_clickbench_q60 1.03 1663.4±22.71µs ? ?/sec 1.00 1612.0±5.14µs ? ?/sec
physical_plan_clickbench_q7 1.00 1329.9±5.73µs ? ?/sec 1.00 1335.0±4.15µs ? ?/sec
physical_plan_clickbench_q8 1.00 1828.3±6.42µs ? ?/sec 1.00 1834.5±7.62µs ? ?/sec
physical_plan_clickbench_q9 1.00 1756.1±4.78µs ? ?/sec 1.01 1777.8±6.62µs ? ?/sec
physical_plan_struct_join_agg_sort 1.00 1193.6±4.05µs ? ?/sec 1.00 1196.1±2.13µs ? ?/sec
physical_plan_tpcds_all 1.00 561.4±3.46ms ? ?/sec 1.01 567.3±2.81ms ? ?/sec
physical_plan_tpch_all 1.00 39.5±0.08ms ? ?/sec 1.01 40.1±0.17ms ? ?/sec
physical_plan_tpch_q1 1.01 1408.1±4.40µs ? ?/sec 1.00 1398.1±2.58µs ? ?/sec
physical_plan_tpch_q10 1.00 2.1±0.00ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
physical_plan_tpch_q11 1.00 2.1±0.00ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
physical_plan_tpch_q12 1.00 1188.5±2.58µs ? ?/sec 1.02 1216.3±3.14µs ? ?/sec
physical_plan_tpch_q13 1.00 947.3±2.52µs ? ?/sec 1.02 968.3±1.99µs ? ?/sec
physical_plan_tpch_q14 1.00 1216.3±2.63µs ? ?/sec 1.03 1250.7±2.45µs ? ?/sec
physical_plan_tpch_q16 1.02 1554.2±2.89µs ? ?/sec 1.00 1527.6±2.26µs ? ?/sec
physical_plan_tpch_q17 1.00 1507.7±3.26µs ? ?/sec 1.00 1504.0±3.81µs ? ?/sec
physical_plan_tpch_q18 1.00 1741.0±4.55µs ? ?/sec 1.00 1735.3±3.27µs ? ?/sec
physical_plan_tpch_q19 1.00 1958.2±4.69µs ? ?/sec 1.00 1959.7±3.36µs ? ?/sec
physical_plan_tpch_q2 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.00ms ? ?/sec
physical_plan_tpch_q20 1.00 1931.9±4.71µs ? ?/sec 1.03 1988.2±16.60µs ? ?/sec
physical_plan_tpch_q21 1.00 2.4±0.00ms ? ?/sec 1.01 2.4±0.00ms ? ?/sec
physical_plan_tpch_q22 1.00 1468.2±3.03µs ? ?/sec 1.00 1472.4±2.39µs ? ?/sec
physical_plan_tpch_q3 1.00 1600.6±4.59µs ? ?/sec 1.02 1630.1±3.15µs ? ?/sec
physical_plan_tpch_q4 1.00 1055.2±6.61µs ? ?/sec 1.01 1069.1±3.02µs ? ?/sec
physical_plan_tpch_q5 1.01 2.2±0.02ms ? ?/sec 1.00 2.2±0.00ms ? ?/sec
physical_plan_tpch_q6 1.00 694.0±1.81µs ? ?/sec 1.01 701.9±1.49µs ? ?/sec
physical_plan_tpch_q7 1.01 2.5±0.01ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
physical_plan_tpch_q8 1.01 3.3±0.03ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
physical_plan_tpch_q9 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
physical_select_aggregates_from_200 1.01 7.4±0.03ms ? ?/sec 1.00 7.3±0.01ms ? ?/sec
physical_select_all_from_1000 1.00 17.0±0.04ms ? ?/sec 1.02 17.3±0.06ms ? ?/sec
physical_select_one_from_700 1.01 643.7±5.75µs ? ?/sec 1.00 637.6±7.37µs ? ?/sec
physical_sorted_union_order_by_10_int64 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.00ms ? ?/sec
physical_sorted_union_order_by_10_uint64 1.03 6.9±0.05ms ? ?/sec 1.00 6.7±0.01ms ? ?/sec
physical_sorted_union_order_by_50_int64 1.02 75.8±0.43ms ? ?/sec 1.00 74.6±0.18ms ? ?/sec
physical_sorted_union_order_by_50_uint64 1.02 257.7±1.79ms ? ?/sec 1.00 252.6±0.90ms ? ?/sec
physical_theta_join_consider_sort 1.00 968.3±3.02µs ? ?/sec 1.03 993.0±1.43µs ? ?/sec
physical_unnest_to_join 1.00 552.7±2.45µs ? ?/sec 1.04 572.6±1.29µs ? ?/sec
physical_window_function_partition_by_12_on_values 1.00 633.3±2.20µs ? ?/sec 1.01 640.5±1.16µs ? ?/sec
physical_window_function_partition_by_30_on_values 1.00 1198.4±9.60µs ? ?/sec 1.01 1214.6±2.78µs ? ?/sec
physical_window_function_partition_by_4_on_values 1.00 395.0±0.90µs ? ?/sec 1.00 396.7±1.43µs ? ?/sec
physical_window_function_partition_by_7_on_values 1.00 484.2±1.61µs ? ?/sec 1.00 483.3±1.19µs ? ?/sec
physical_window_function_partition_by_8_on_values 1.02 519.6±1.85µs ? ?/sec 1.00 509.1±1.40µs ? ?/sec
with_param_values_many_columns 1.00 368.3±1.80µs ? ?/sec 1.02 376.0±1.96µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sql_planner — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2025.4s |
| Peak memory | 139.6 MiB |
| Avg memory | 90.1 MiB |
| CPU user | 1972.4s |
| CPU sys | 1.5s |
| Peak spill | 0 B |
**sql_planner — branch**
| Metric | Value |
|--------|-------|
| Wall time | 2125.5s |
| Peak memory | 140.0 MiB |
| Avg memory | 94.3 MiB |
| CPU user | 2104.5s |
| CPU sys | 1.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6223916978
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26428?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.00000%` with `93 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.89%. Comparing base ([`5009f42`](https://app.codecov.io/gh/apache/datafusion/commit/5009f42e4a24d231580222e5628c1e203f07614b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cd299a2`](https://app.codecov.io/gh/apache/datafusion/commit/cd299a251a76a20c2a5acc7e38921bba56852183?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 11 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26428?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 84.75% | [19 Missing and 31 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-plan/src/materialized\_cte.rs](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fmaterialized_cte.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9tYXRlcmlhbGl6ZWRfY3RlLnJz) | 86.00% | [5 Missing and 30 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/core/src/datasource/view\_test.rs](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fdatasource%2Fview_test.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9kYXRhc291cmNlL3ZpZXdfdGVzdC5ycw==) | 80.95% | [0 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26428?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26428 +/- ##
==========================================
+ Coverage 82.87% 82.89% +0.01%
==========================================
Files 1153 1153
Lines 457101 458065 +964
Branches 457101 458065 +964
==========================================
+ Hits 378818 379707 +889
- Misses 55268 55273 +5
- Partials 23015 23085 +70
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26428?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "adriangb (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangb commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6569850863
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "github-actions[bot] (via GitHub) via github" <gi...@datafusion.apache.org>.
github-actions[bot] commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6569580059
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning 636590bf4f4b4f792d3f2fdacedc5fee0ba54950
Building datafusion v55.1.0 (current)
Built [ 41.762s] (current)
Parsing datafusion v55.1.0 (current)
Parsed [ 0.026s] (current)
Building datafusion v55.1.0 (baseline)
Built [ 13.689s] (baseline)
Parsing datafusion v55.1.0 (baseline)
Parsed [ 0.026s] (baseline)
Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.462s] 229 checks: 229 pass, 31 skip
Summary no semver update required
Finished [ 57.970s] datafusion
Building datafusion-physical-plan v55.1.0 (current)
Built [ 12.425s] (current)
Parsing datafusion-physical-plan v55.1.0 (current)
Parsed [ 0.163s] (current)
Building datafusion-physical-plan v55.1.0 (baseline)
Built [ 9.052s] (baseline)
Parsing datafusion-physical-plan v55.1.0 (baseline)
Parsed [ 0.168s] (baseline)
Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.510s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip
--- failure method_parameter_count_changed: pub method parameter count changed ---
Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/method_parameter_count_changed.ron
Failed in:
datafusion_physical_plan::materialized_cte::MaterializedCteScanExec::with_body_statistics takes 1 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks-shared/semver-checks/git-636590bf4f4b4f792d3f2fdacedc5fee0ba54950/e2a3e96438e593e600b6e79c1af4aa2a321a8c97/datafusion/physical-plan/src/materialized_cte.rs:722, but now takes 2 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/materialized_cte.rs:787
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 23.517s] datafusion-physical-plan
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "github-actions[bot] (via GitHub) via github" <gi...@datafusion.apache.org>.
github-actions[bot] commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6934693574
Marked as draft after 14 days without activity: merge conflicts.
Please address these items and mark **Ready for review** when ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "asolimando (via GitHub) via github" <gi...@datafusion.apache.org>.
asolimando commented on code in PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#discussion_r4437457791
##########
datafusion/physical-plan/src/materialized_cte.rs:
##########
@@ -816,10 +929,20 @@ impl ExecutionPlan for MaterializedCteScanExec {
);
};
let mut output = buffer.materialize(context)?;
- let output_partitions = self.cache.partitioning.partition_count();
+ let output_partitions = self.partition_count();
+ let mirrors_body = self.mirrors_body;
+ let name = self.name.clone();
let baseline = BaselineMetrics::new(&self.metrics, partition);
let stream = futures::stream::once(async move {
let output = std::future::poll_fn(|cx| output.get_shared(cx)).await?;
+ // The ordering and the hash partitioning of the scan are only
+ // correct when one scan partition replays one body partition.
+ if mirrors_body && output.partitions.len() != output_partitions {
Review Comment:
Here we check only the partition count, but if you have a body with the same count but a different ordering or hash keys, it goes through with wrong results. I have checked with a variant of `mirror_scan_rejects_other_body_partition_count`: same buffer, unsorted body, the scan still reports `[a ASC]` and returns `[3, 1, 2]`.
I don't think the problem can reproduce in DF core, but projects doing post-planning rewrites like Ballista or datafusion distributed could, we should probably handle this to be on the safe side.
A cheap fix would be checking the buffer's body in `execute` has the same hash partitioning, and an ordering that satisfies the scan's.
If you prefer not to fix it, I think we should at least put a note on `MaterializedCteExec::new`'s documentation saying the body must keep the copied properties.
WDYT?
##########
datafusion/physical-plan/src/materialized_cte.rs:
##########
Review Comment:
This points to the issue this PR closes, but statistics are still only known for single-partition scans. Shall we file a new issue and repoint it?
##########
datafusion/core/src/physical_planner.rs:
##########
@@ -317,125 +321,344 @@ fn renumber_materialized_ctes(plan: &LogicalPlan) -> Result<Option<LogicalPlan>>
}
/// Points every [`MaterializedCteScanExec`] at the buffer of the
-/// [`MaterializedCteExec`] with the same id, and gives it the statistics of
-/// the CTE body.
+/// [`MaterializedCteExec`] with the same id.
///
/// [`renumber_materialized_ctes`] makes the ids unique before the physical
/// plan is created.
///
-/// A scan that is already bound was planned and bound by a separate planning
-/// run, such as the `scan` of a [`ViewTable`], and is not bound again. Binding
-/// it again would count its partitions twice in the buffer.
+/// A scan that is already bound comes from a separate planning run, such as
+/// the `scan` of a [`ViewTable`], and is bound again. That run can have given
+/// it the ordering and the partitioning of a body that is optimized again
+/// now, and the new scan does not have them.
+/// [`register_materialized_ctes`] sets the partition counts of the scans of
+/// the final plan in the buffer.
///
/// Scans are planned before the node that owns their CTE, and a scan inside a
/// scalar subquery is planned in a separate subtree, so the binding is done
/// on the whole initial plan.
///
/// [`ViewTable`]: crate::datasource::ViewTable
-struct MaterializedCteBinder {
- ctes: HashMap<u64, Arc<dyn ExecutionPlan>>,
- /// The statistics of the CTE bodies. `None` while they are computed.
- statistics: HashMap<u64, Option<Arc<Statistics>>>,
- context: StatisticsContext,
+fn bind_materialized_cte_scans(
+ plan: Arc<dyn ExecutionPlan>,
+ ctes: &HashMap<u64, Arc<dyn ExecutionPlan>>,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ plan.transform_up(|node| {
+ let Some(scan) = node.downcast_ref::<MaterializedCteScanExec>() else {
+ return Ok(Transformed::no(node));
+ };
+ let Some(cte) = ctes.get(&scan.id()) else {
+ if scan.is_bound() {
+ return Ok(Transformed::no(node));
+ }
+ return internal_err!("MaterializedCteScanExec {} has no CTE", scan.id());
+ };
+ let scan = scan.bind(Arc::clone(as_materialized_cte(cte)?.buffer()));
+ Ok(Transformed::yes(Arc::new(scan) as Arc<dyn ExecutionPlan>))
+ })
+ .map(|t| t.data)
}
-impl MaterializedCteBinder {
- fn new(plan: &Arc<dyn ExecutionPlan>, session_state: &dyn Session) -> Result<Self> {
- let mut ctes = HashMap::new();
- plan.apply(|node| {
- if let Some(cte) = node.downcast_ref::<MaterializedCteExec>()
- && ctes.insert(cte.buffer().id(), Arc::clone(node)).is_some()
+/// The [`MaterializedCteExec`] nodes of `plan`, by id.
+fn materialized_ctes(
+ plan: &Arc<dyn ExecutionPlan>,
+) -> Result<HashMap<u64, Arc<dyn ExecutionPlan>>> {
+ let mut ctes = HashMap::new();
+ plan.apply(|node| {
+ if let Some(cte) = node.downcast_ref::<MaterializedCteExec>()
+ && ctes.insert(cte.buffer().id(), Arc::clone(node)).is_some()
+ {
+ return internal_err!(
+ "Two MaterializedCteExec nodes have the id {}",
+ cte.buffer().id()
+ );
+ }
+ Ok(TreeNodeRecursion::Continue)
+ })?;
+ Ok(ctes)
+}
+
+fn as_materialized_cte(plan: &Arc<dyn ExecutionPlan>) -> Result<&MaterializedCteExec> {
+ match plan.downcast_ref::<MaterializedCteExec>() {
+ Some(cte) => Ok(cte),
+ None => internal_err!("Expected a MaterializedCteExec"),
+ }
+}
+
+/// The ids of `ctes`, in an order in which each CTE comes after the CTEs
+/// that its body reads or defines.
+fn materialized_cte_order(
+ ctes: &HashMap<u64, Arc<dyn ExecutionPlan>>,
+) -> Result<Vec<u64>> {
+ fn visit(
+ id: u64,
+ ctes: &HashMap<u64, Arc<dyn ExecutionPlan>>,
+ visiting: &mut HashSet<u64>,
+ order: &mut Vec<u64>,
+ ) -> Result<()> {
+ if order.contains(&id) {
+ return Ok(());
+ }
+ if !visiting.insert(id) {
+ return internal_err!("The body of MaterializedCteExec {id} reads itself");
+ }
+ let mut reads = vec![];
+ as_materialized_cte(&ctes[&id])?.body().apply(|node| {
+ if let Some(scan) = node.downcast_ref::<MaterializedCteScanExec>()
+ && ctes.contains_key(&scan.id())
{
- return internal_err!(
- "Two MaterializedCteExec nodes have the id {}",
- cte.buffer().id()
- );
+ reads.push(scan.id());
+ }
+ if let Some(cte) = node.downcast_ref::<MaterializedCteExec>() {
+ reads.push(cte.buffer().id());
}
Ok(TreeNodeRecursion::Continue)
})?;
- let context = match session_state.statistics_registry() {
- Some(registry) => StatisticsContext::new_with_registry(registry.clone()),
- None => StatisticsContext::new(),
- };
- Ok(Self {
- ctes,
- statistics: HashMap::new(),
- context,
- })
+ for read in reads {
+ visit(read, ctes, visiting, order)?;
+ }
+ order.push(id);
+ Ok(())
}
- fn cte(&self, id: u64) -> Option<&MaterializedCteExec> {
- self.ctes
- .get(&id)
- .and_then(|cte| cte.downcast_ref::<MaterializedCteExec>())
+ let mut ids: Vec<u64> = ctes.keys().copied().collect();
+ ids.sort_unstable();
+ let mut visiting = HashSet::new();
+ let mut order = vec![];
+ for id in ids {
+ visit(id, ctes, &mut visiting, &mut order)?;
}
+ Ok(order)
+}
- /// Give every scan in `plan` whose CTE is in `self.ctes` the statistics
- /// of the CTE body. If `bind` is true, also bind the scans that are not
- /// bound.
- fn bind(
- &mut self,
- plan: Arc<dyn ExecutionPlan>,
- bind: bool,
- ) -> Result<Arc<dyn ExecutionPlan>> {
- plan.transform_up(|node| {
- let Some(scan) = node.downcast_ref::<MaterializedCteScanExec>() else {
- return Ok(Transformed::no(node));
- };
- let Some(cte) = self.cte(scan.id()) else {
- if scan.is_bound() {
- return Ok(Transformed::no(node));
+/// A CTE body after the optimizers ran on it.
+struct OptimizedCteBody {
+ plan: Arc<dyn ExecutionPlan>,
+ statistics: Arc<Statistics>,
+}
+
+/// Prepare `plan` for the optimizers, after the bodies in `bodies` are
+/// optimized:
+///
+/// - Each scan whose CTE is in `bodies` gets the statistics, the partition
+/// count, the output ordering and the hash partitioning of that body.
+/// - Each [`MaterializedCteExec`] whose body is in `bodies` becomes a
+/// [`FrozenMaterializedCteExec`], so that the optimizers do not see or
+/// change the body again.
+fn freeze_materialized_ctes(
+ plan: Arc<dyn ExecutionPlan>,
+ bodies: &HashMap<u64, OptimizedCteBody>,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ plan.transform_up(|node| {
+ if let Some(scan) = node.downcast_ref::<MaterializedCteScanExec>()
+ && let Some(body) = bodies.get(&scan.id())
+ {
+ let scan = scan.with_body_statistics(Arc::clone(&body.statistics));
+ let scan = scan
+ .with_body_properties(body.plan.as_ref())
+ .unwrap_or(scan);
+ return Ok(Transformed::yes(Arc::new(scan) as Arc<dyn ExecutionPlan>));
+ }
+ if let Some(cte) = node.downcast_ref::<MaterializedCteExec>()
+ && let Some(body) = bodies.get(&cte.buffer().id())
+ {
+ let continuation = Arc::clone(node.children()[1]);
+ let frozen = FrozenMaterializedCteExec::new(
+ Arc::clone(&body.plan),
+ continuation,
+ Arc::clone(cte.buffer()),
+ );
+ return Ok(Transformed::yes(Arc::new(frozen) as Arc<dyn ExecutionPlan>));
+ }
+ Ok(Transformed::no(node))
+ })
+ .map(|t| t.data)
+}
+
+/// Turn each [`FrozenMaterializedCteExec`] of `plan` back into a
+/// [`MaterializedCteExec`].
+fn thaw_materialized_ctes(
+ plan: Arc<dyn ExecutionPlan>,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ plan.transform_up(|node| {
+ let Some(frozen) = node.downcast_ref::<FrozenMaterializedCteExec>() else {
+ return Ok(Transformed::no(node));
+ };
+ let cte = MaterializedCteExec::new(
+ Arc::clone(&frozen.body),
+ Arc::clone(&frozen.continuation),
+ Arc::clone(&frozen.buffer),
+ );
+ Ok(Transformed::yes(Arc::new(cte) as Arc<dyn ExecutionPlan>))
+ })
+ .map(|t| t.data)
+}
+
+mod frozen_cte {
+ use std::fmt;
+ use std::sync::Arc;
+
+ use datafusion_common::config::ConfigOptions;
+ use datafusion_common::tree_node::TreeNodeRecursion;
+ use datafusion_common::{Result, Statistics, internal_err};
+ use datafusion_execution::{SendableRecordBatchStream, TaskContext};
+ use datafusion_physical_expr::PhysicalExpr;
+ use datafusion_physical_plan::execution_plan::{CardinalityEffect, PlanProperties};
+ use datafusion_physical_plan::filter_pushdown::{
+ ChildFilterDescription, ChildPushdownResult, FilterDescription,
+ FilterPushdownPhase, FilterPushdownPropagation,
+ };
+ use datafusion_physical_plan::materialized_cte::MaterializedCteBuffer;
+ use datafusion_physical_plan::statistics::StatisticsArgs;
+ use datafusion_physical_plan::{DisplayAs, DisplayFormatType, ExecutionPlan};
+
+ #[cfg(doc)]
+ use datafusion_physical_plan::materialized_cte::MaterializedCteExec;
+
+ /// A [`MaterializedCteExec`] whose body is already optimized, while the
+ /// optimizers run on the rest of the plan.
+ ///
+ /// The body is not a child, so the optimizers cannot change it, and the
+ /// properties that the scans copied from it stay true. For the continuation,
+ /// this node acts like a [`MaterializedCteExec`].
+ #[derive(Debug)]
+ pub(super) struct FrozenMaterializedCteExec {
+ pub(super) body: Arc<dyn ExecutionPlan>,
+ pub(super) continuation: Arc<dyn ExecutionPlan>,
+ pub(super) buffer: Arc<MaterializedCteBuffer>,
+ }
+
+ impl FrozenMaterializedCteExec {
+ pub(super) fn new(
+ body: Arc<dyn ExecutionPlan>,
+ continuation: Arc<dyn ExecutionPlan>,
+ buffer: Arc<MaterializedCteBuffer>,
+ ) -> Self {
+ Self {
+ body,
+ continuation,
+ buffer,
+ }
+ }
+ }
+
+ impl DisplayAs for FrozenMaterializedCteExec {
+ fn fmt_as(&self, t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {
+ match t {
+ DisplayFormatType::Default | DisplayFormatType::Verbose => {
+ write!(f, "MaterializedCteExec: name={}", self.buffer.name())
Review Comment:
I understand we don't want to leak the freezing of CTEs in plans, but plan display is the quickest debugging tool: in `EXPLAIN VERBOSE` the body disappears after the first rule while the node still shows as a regular `MaterializedCteExec`, so it looks like the body was dropped, which can be confusing.
Is there a way we can mark the node as frozen, or still show its body, somehow?
##########
datafusion/physical-plan/src/materialized_cte.rs:
##########
@@ -737,13 +786,67 @@ impl MaterializedCteScanExec {
id: self.id,
name: self.name.clone(),
schema: Arc::clone(&self.schema),
+ partitions: self.partitions,
+ mirrors_body: self.mirrors_body,
statistics: Some(statistics),
buffer: self.buffer.clone(),
metrics: ExecutionPlanMetricsSet::new(),
cache: Arc::clone(&self.cache),
}
}
+ /// Give the scan the partition count of `body`, its output ordering and
+ /// its hash partitioning. Partition `p` of the scan then replays
+ /// partition `p` of the body.
+ ///
+ /// Returns `None` when `body` has no output ordering and no hash
+ /// partitioning, or when its schema is not the schema of the scan.
+ ///
+ /// The caller must make sure that the body of the plan that is executed
+ /// is `body`, or has the same properties.
+ pub fn with_body_properties(&self, body: &dyn ExecutionPlan) -> Option<Self> {
+ let body_schema = body.schema();
+ let same_columns = self.schema.fields().len() == body_schema.fields().len()
+ && self
+ .schema
+ .fields()
+ .iter()
+ .zip(body_schema.fields())
+ .all(|(a, b)| a.name() == b.name() && a.data_type() == b.data_type());
+ if !same_columns {
+ return None;
+ }
+ let ordering = body.output_ordering().cloned();
Review Comment:
Nit: this keeps only the first ordering, and drops the equivalences and constants of the body, but they would still be valid for the scan. Copying the body’s equivalence properties (with the scan schema) would preserve them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6475276065
I started checking and it looks good to me, there are a couple of points to discuss but the direction is good. I was a bit optimistic I could review by EOD today, but it's more realistic I will manage to provide a proper review by the end of this week, thanks for the ping, Adrian!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6457240086
cc @asolimando
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: report the ordering and partitioning of materialized CTE scans [datafusion]
Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26428:
URL: https://github.com/apache/datafusion/pull/26428#issuecomment-6569895562
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26428#issuecomment-6569850863)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6569850863-4228-fw6fh 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing mcte-body-first-properties (4824866f4a324b75674127d1646070fea4a71d0c) to 985bbbf (merge-base) [diff](https://github.com/apache/datafusion/compare/985bbbf60e14644d8945dce9342f6dd20a428f1a..4824866f4a324b75674127d1646070fea4a71d0c)
<details><summary>Run configuration</summary>
```yaml
run benchmark sql_planner
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org