You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/10/19 03:30:43 UTC
[PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
kosiew opened a new pull request, #26487:
URL: https://github.com/apache/datafusion/pull/26487
## Which issue does this PR close?
- Part of #23393
## Rationale for this change
Symmetric hash joins use memory reservations to track retained execution state. Previously, `ExprIntervalGraph::size()` estimated memory using live node and edge counts, which could underreport memory retained by the graph after nodes or edges were removed. It also omitted heap allocations owned by interval bounds.
This change improves reservation accuracy by accounting for retained graph backing capacity and interval-owned heap allocations, while excluding shared physical-expression allocations.
## What changes are included in this PR?
- Replace live node and edge count estimates with capacity-based accounting using the underlying `petgraph::StableGraph` node and edge storage types.
- Include heap allocations owned by interval bounds using `DFHeapSize` and a shared `DFHeapSizeCtx` to avoid double-counting shared allocations.
- Exclude shared `Arc<dyn PhysicalExpr>` pointees from graph-owned memory accounting.
- Add regression coverage verifying that `SymmetricHashJoinStream` reservations reflect the graph's reported size after construction, interval updates, and removal.
## Are these changes tested?
Three regression tests are added:
- `graph_size_counts_retained_capacity`: Verifies empty and grown graph accounting, including retained capacity after node removal and graph clearing.
- `graph_size_counts_interval_heaps_not_expression_pointees`: Verifies UTF-8 interval heap accounting, deduplication of shared timestamp timezone allocations, and exclusion of shared physical-expression pointees.
- `stream_reserves_retained_interval_graph`: Verifies that stream reservations and memory metrics account for the retained interval graph and are updated when the graph changes or is removed.
The tests use deterministic capacity and size assertions without relying on global allocator measurements.
## Are there any user-facing changes?
No changes to query results or public APIs are intended. Memory reservations for symmetric hash joins may now more accurately reflect retained interval graph allocations, potentially affecting memory-limit enforcement.
No documentation changes are expected.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6517538044
🚀
@alamb
Thank you for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507767874
run benchmark sql_planner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6259607146
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.26984%` with `11 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.89%. Comparing base ([`fe612e2`](https://app.codecov.io/gh/apache/datafusion/commit/fe612e2da212016905d600eb0dd689910c0899b7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`59148b1`](https://app.codecov.io/gh/apache/datafusion/commit/59148b108a98155f3be2b0966ca7a8b04baef0ea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ion/physical-plan/src/joins/symmetric\_hash\_join.rs](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fsymmetric_hash_join.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9zeW1tZXRyaWNfaGFzaF9qb2luLnJz) | 73.33% | [0 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...atafusion/physical-expr/src/intervals/cp\_solver.rs](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fintervals%2Fcp_solver.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9pbnRlcnZhbHMvY3Bfc29sdmVyLnJz) | 96.87% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26487?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26487 +/- ##
==========================================
- Coverage 82.89% 82.89% -0.01%
==========================================
Files 1154 1154
Lines 458022 458179 +157
Branches 458022 458179 +157
==========================================
+ Hits 379683 379811 +128
- Misses 55303 55315 +12
- Partials 23036 23053 +17
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26487?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #26487:
URL: https://github.com/apache/datafusion/pull/26487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#discussion_r4417617871
##########
datafusion/physical-expr/src/intervals/cp_solver.rs:
##########
@@ -817,6 +830,108 @@ mod tests {
use rand::{Rng, SeedableRng};
use rstest::*;
+ fn expected_graph_backing_size(graph: &ExprIntervalGraph) -> usize {
Review Comment:
this is a lot of lines of test -- I wonder if there is some way to reduce the replication / make some fixtures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507817514
Benchmark for [this request](https://github.com/apache/datafusion/pull/26487#issuecomment-6507767874) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `sql_planner`
<details><summary>Runner log (last 40 lines)</summary>
```
2026-10-29T20:39:43.953683Z INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26487, benchmarks=sql_planner
2026-10-29T20:39:44.013544Z INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-29T20:39:44.013567Z INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-29T20:39:54.028932Z INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26487/head:memcalc-15-23393", "main"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:14.034021Z INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "memcalc-15-23393"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:19.035812Z INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:24.037804Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:
stderr:
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507827945
Benchmark for [this request](https://github.com/apache/datafusion/pull/26487#issuecomment-6507765304) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `sql_parser`
<details><summary>Runner log (last 40 lines)</summary>
```
2026-10-29T20:40:20.046061Z INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26487, benchmarks=sql_parser
2026-10-29T20:40:20.105072Z INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-29T20:40:20.105095Z INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-29T20:40:25.116029Z INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26487/head:memcalc-15-23393", "main"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:45.121280Z INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "memcalc-15-23393"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:50.124681Z INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-29T20:40:55.126141Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:
stderr:
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#issuecomment-6507765304
run benchmark sql_parser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix retained memory accounting for ExprIntervalGraph [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #26487:
URL: https://github.com/apache/datafusion/pull/26487#discussion_r4421429975
##########
datafusion/physical-expr/src/intervals/cp_solver.rs:
##########
@@ -817,6 +830,108 @@ mod tests {
use rand::{Rng, SeedableRng};
use rstest::*;
+ fn expected_graph_backing_size(graph: &ExprIntervalGraph) -> usize {
Review Comment:
Added shared graph/node setup helpers in cp_solver - https://github.com/apache/datafusion/pull/26487/commits/2a77e6d31fb791df1344b9176850cd0bd1ab1c37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org