You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/10/19 03:30:43 UTC

[PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

kosiew opened a new pull request, #26487:
URL: https://github.com/apache/datafusion/pull/26487

   ## Which issue does this PR close?
   
   - Part of #23393
   
   ## Rationale for this change
   
   Symmetric hash joins use memory reservations to track retained execution state. Previously, `ExprIntervalGraph::size()` estimated memory using live node and edge counts, which could underreport memory retained by the graph after nodes or edges were removed. It also omitted heap allocations owned by interval bounds.
   
   This change improves reservation accuracy by accounting for retained graph backing capacity and interval-owned heap allocations, while excluding shared physical-expression allocations.
   
   ## What changes are included in this PR?
   
   - Replace live node and edge count estimates with capacity-based accounting using the underlying `petgraph::StableGraph` node and edge storage types.
   - Include heap allocations owned by interval bounds using `DFHeapSize` and a shared `DFHeapSizeCtx` to avoid double-counting shared allocations.
   - Exclude shared `Arc<dyn PhysicalExpr>` pointees from graph-owned memory accounting.
   - Add regression coverage verifying that `SymmetricHashJoinStream` reservations reflect the graph's reported size after construction, interval updates, and removal.
   
   ## Are these changes tested?
   
   Three regression tests are added:
   
   - `graph_size_counts_retained_capacity`: Verifies empty and grown graph accounting, including retained capacity after node removal and graph clearing.
   - `graph_size_counts_interval_heaps_not_expression_pointees`: Verifies UTF-8 interval heap accounting, deduplication of shared timestamp timezone allocations, and exclusion of shared physical-expression pointees.
   - `stream_reserves_retained_interval_graph`: Verifies that stream reservations and memory metrics account for the retained interval graph and are updated when the graph changes or is removed.
   
   The tests use deterministic capacity and size assertions without relying on global allocator measurements.
   
   ## Are there any user-facing changes?
   
   No changes to query results or public APIs are intended. Memory reservations for symmetric hash joins may now more accurately reflect retained interval graph allocations, potentially affecting memory-limit enforcement.
   
   No documentation changes are expected.
   
   ## LLM-generated code disclosure
   
   This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6517538044

   🚀 
   @alamb 
   Thank you for your review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507767874

   run benchmark sql_planner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6259607146

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `91.26984%` with `11 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.89%. Comparing base ([`fe612e2`](https://app.codecov.io/gh/apache/datafusion/commit/fe612e2da212016905d600eb0dd689910c0899b7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`59148b1`](https://app.codecov.io/gh/apache/datafusion/commit/59148b108a98155f3be2b0966ca7a8b04baef0ea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ion/physical-plan/src/joins/symmetric\_hash\_join.rs](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fsymmetric_hash_join.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9zeW1tZXRyaWNfaGFzaF9qb2luLnJz) | 73.33% | [0 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...atafusion/physical-expr/src/intervals/cp\_solver.rs](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fintervals%2Fcp_solver.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9pbnRlcnZhbHMvY3Bfc29sdmVyLnJz) | 96.87% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #26487      +/-   ##
   ==========================================
   - Coverage   82.89%   82.89%   -0.01%     
   ==========================================
     Files        1154     1154              
     Lines      458022   458179     +157     
     Branches   458022   458179     +157     
   ==========================================
   + Hits       379683   379811     +128     
   - Misses      55303    55315      +12     
   - Partials    23036    23053      +17     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #26487:
URL: https://github.com/apache/datafusion/pull/26487


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#discussion_r4417617871


##########
datafusion/physical-expr/src/intervals/cp_solver.rs:
##########
@@ -817,6 +830,108 @@ mod tests {
     use rand::{Rng, SeedableRng};
     use rstest::*;
 
+    fn expected_graph_backing_size(graph: &ExprIntervalGraph) -> usize {

Review Comment:
   this is a lot of lines of test -- I wonder if there is some way to reduce the replication / make some fixtures?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507817514

   Benchmark for [this request](https://github.com/apache/datafusion/pull/26487#issuecomment-6507767874) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
   
   Benchmarks requested: `sql_planner`
   
   <details><summary>Runner log (last 40 lines)</summary>
   
   ```
   2026-10-29T20:39:43.953683Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26487, benchmarks=sql_planner
   2026-10-29T20:39:44.013544Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
   2026-10-29T20:39:44.013567Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
   2026-10-29T20:39:54.028932Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26487/head:memcalc-15-23393", "main"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:14.034021Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "memcalc-15-23393"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:19.035812Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:24.037804Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
   stdout:
   
   stderr:
   ```
   
   </details>
   
   <details><summary>Kubernetes message</summary>
   
   ```
   Job has reached the specified backoff limit
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507827945

   Benchmark for [this request](https://github.com/apache/datafusion/pull/26487#issuecomment-6507765304) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
   
   Benchmarks requested: `sql_parser`
   
   <details><summary>Runner log (last 40 lines)</summary>
   
   ```
   2026-10-29T20:40:20.046061Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26487, benchmarks=sql_parser
   2026-10-29T20:40:20.105072Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
   2026-10-29T20:40:20.105095Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
   2026-10-29T20:40:25.116029Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26487/head:memcalc-15-23393", "main"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:45.121280Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "memcalc-15-23393"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:50.124681Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
   2026-10-29T20:40:55.126141Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
   stdout:
   
   stderr:
   ```
   
   </details>
   
   <details><summary>Kubernetes message</summary>
   
   ```
   Job has reached the specified backoff limit
   ```
   
   </details>
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507765304

   run benchmark sql_parser


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#discussion_r4421429975


##########
datafusion/physical-expr/src/intervals/cp_solver.rs:
##########
@@ -817,6 +830,108 @@ mod tests {
     use rand::{Rng, SeedableRng};
     use rstest::*;
 
+    fn expected_graph_backing_size(graph: &ExprIntervalGraph) -> usize {

Review Comment:
   Added shared graph/node setup helpers in cp_solver - https://github.com/apache/datafusion/pull/26487/commits/2a77e6d31fb791df1344b9176850cd0bd1ab1c37
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org