You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/10/13 17:30:04 UTC

[PR] feat: release the output of a materialized CTE when the last scan finishes [datafusion]

adriangb opened a new pull request, #26272:
URL: https://github.com/apache/datafusion/pull/26272

   ## Which issue does this PR close?
   
   - Closes apache/datafusion#26233.
   
   Stacked on apache/datafusion#25580 and on #26270 (apache/datafusion#26229). Only the last commit is new.
   
   ## Rationale for this change
   
   The buffered output of a materialized CTE (in-memory batches, memory reservations, spill files) was held until the end of the run. A CTE is often read early, and the rest of the query then runs while the buffer holds memory and disk space that nothing reads again.
   
   ```sql
   WITH c AS MATERIALIZED (SELECT value AS v FROM generate_series(1, 20000))
   SELECT count(*) FROM c a JOIN c b ON a.v = b.v
   WHERE a.v <= (SELECT max(v) FROM c);
   ```
   
   Before: after the output stream ends, the memory pool still has 196896 bytes reserved for `MaterializedCte[c]` until the `TaskContext` is dropped. After: the reservation is 0 when the last of the three scans finishes.
   
   ## What changes are included in this PR?
   
   - The output of a CTE in a run counts its readers. It is released when each partition of each bound scan was executed in the run, and each of these executions has finished or was dropped.
   - The count is of the partitions that can still be executed, not of the open streams. Thus the output is not released between two scans that are executed one after the other.
   - A scan stream that is dropped before its end (for example below a `LIMIT`) counts as finished. If all scan streams are dropped before the body completes, the body is cancelled.
   - A scan partition that is never executed keeps the output until the run ends, as before.
   - A scan partition that is executed after the release computes the body again. It never reads an empty output.
   - `RecursiveQueryExec` holds the outputs of the CTEs that its recursive term reads and does not declare, so these are computed one time for the whole query.
   
   Not included: the release of each body partition when all scans have read it.
   
   Known limits:
   
   - If an operator other than `RecursiveQueryExec` executes a scan partition again after the release, the body is computed again. For a volatile body, the second read then has different rows. No operator in the repository does this.
   - The hold of `RecursiveQueryExec` ends when its stream is dropped, not when the stream ends.
   
   ## Are these changes tested?
   
   Yes. Unit tests check that the pool reservation is 0 and the spill files are deleted after the last scan partition finishes while the plan and the `TaskContext` are alive, and cover streams dropped early, body cancellation, a partition that is never executed, a scan after the release, and the hold. A test in `datafusion/core/tests/sql/select.rs` checks the release for a planned SQL query. Four existing unit tests asserted the old lifetime and now assert the new one.
   
   ## Are there any user-facing changes?
   
   No API change. Memory and spill files of a materialized CTE are released earlier. A plan that is executed two times with the same `TaskContext` computes the body again in the second execution, if the first has finished.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: release the output of a materialized CTE when the last scan finishes [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26272:
URL: https://github.com/apache/datafusion/pull/26272#issuecomment-6156729731

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26272?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `86.23794%` with `214 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.85%. Comparing base ([`2c568b4`](https://app.codecov.io/gh/apache/datafusion/commit/2c568b478b3aff18de52609c7237f1c3f494d5b1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1e47822`](https://app.codecov.io/gh/apache/datafusion/commit/1e478226557dec12a5a6e7310639c630a390b8f1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26272?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/physical-plan/src/materialized\_cte.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fmaterialized_cte.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9tYXRlcmlhbGl6ZWRfY3RlLnJz) | 85.67% | [60 Missing and 88 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...tafusion/expr/src/logical\_plan/materialized\_cte.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fmaterialized_cte.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vbWF0ZXJpYWxpemVkX2N0ZS5ycw==) | 84.95% | [17 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 88.88% | [8 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/core/src/datasource/view\_test.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fdatasource%2Fview_test.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9kYXRhc291cmNlL3ZpZXdfdGVzdC5ycw==) | 76.19% | [3 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/optimizer/src/decorrelate.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fdecorrelate.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2RlY29ycmVsYXRlLnJz) | 62.96% | [9 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/optimizer/src/analyzer/type\_coercion.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Ftype_coercion.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2FuYWx5emVyL3R5cGVfY29lcmNpb24ucnM=) | 92.59% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/execution/src/task.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fexecution%2Fsrc%2Ftask.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leGVjdXRpb24vc3JjL3Rhc2sucnM=) | 94.11% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/sql/src/query.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Fquery.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3F1ZXJ5LnJz) | 81.81% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/expr/src/logical\_plan/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | 75.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/optimizer/src/push\_down\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fpush_down_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3B1c2hfZG93bl9maWx0ZXIucnM=) | 91.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | ... and [1 more](https://app.codecov.io/gh/apache/datafusion/pull/26272?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #26272      +/-   ##
   ==========================================
   + Coverage   82.83%   82.85%   +0.02%     
   ==========================================
     Files        1149     1151       +2     
     Lines      452798   454338    +1540     
     Branches   452798   454338    +1540     
   ==========================================
   + Hits       375082   376455    +1373     
   - Misses      54893    54942      +49     
   - Partials    22823    22941     +118     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26272?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: release the output of a materialized CTE when the last scan finishes [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #26272:
URL: https://github.com/apache/datafusion/pull/26272#issuecomment-6241862173

   Superseded by #26456, which streams the output of a materialized CTE to its scans and includes the release of the output when the last scan finishes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: release the output of a materialized CTE when the last scan finishes [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed pull request #26272: feat: release the output of a materialized CTE when the last scan finishes
URL: https://github.com/apache/datafusion/pull/26272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org