You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/10/07 07:14:28 UTC
[I] Account for retained sort-key capacity [datafusion]
kosiew opened a new issue, #26099:
URL: https://github.com/apache/datafusion/issues/26099
Parent: #23393
Context: [PR comment](https://github.com/apache/datafusion/pull/26074#discussion_r4198007007)
## Goal
Make partial group-ordering memory accounting include the retained allocation
for its `Vec<ScalarValue>` sort key.
## Invariant / contract
`GroupOrderingPartial::heap_size()` reports the retained heap allocations owned
by its state. For an in-progress state, this includes the sort-key vector's
allocated capacity and heap allocations owned by its initialized scalar
elements, each exactly once. It does not add an inline descriptor already
charged by `GroupOrdering::size()`.
## Tasks
- [ ] Replace the live-element-only sort-key accounting in
`State::heap_size()` with capacity-aware `DFHeapSize` accounting, following
the grouped `COUNT` accumulator's established approach.
- [ ] Add a deterministic test with spare `Vec<ScalarValue>` capacity and a
heap-owning scalar, proving spare slots and initialized scalar payloads are
accounted for without relying on allocator measurements.
- [ ] Preserve the existing ownership boundary: do not charge inline
descriptors or allocations owned by other ordering state again.
## Acceptance criteria
- [ ] An in-progress sort key with spare vector capacity reports that capacity
even when the vector's live length is smaller.
- [ ] Heap allocations owned by initialized `ScalarValue`s remain included.
- [ ] Existing start/complete and ordering-state accounting behavior remains
unchanged.
## Tests / verification
- Add or update the focused `GroupOrderingPartial` tests in
`datafusion/physical-plan/src/aggregates/order/partial.rs`.
- Run `cargo test -p datafusion-physical-plan`.
- Run `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` for this
Markdown change.
## Dependencies / blockers
Keep the descriptor accounting established by the `GroupOrdering` owner
separate from this retained-capacity fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Account for retained sort-key capacity [datafusion]
Posted by "mohitgurav20 (via GitHub)" <gi...@apache.org>.
mohitgurav20 commented on issue #26099:
URL: https://github.com/apache/datafusion/issues/26099#issuecomment-6044744770
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Account for retained sort-key capacity [datafusion]
Posted by "saadtajwar (via GitHub)" <gi...@apache.org>.
saadtajwar commented on issue #26099:
URL: https://github.com/apache/datafusion/issues/26099#issuecomment-6475257902
Sounds good! Feel free to tag me for a review when ready :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Account for retained sort-key capacity [datafusion]
Posted by "saadtajwar (via GitHub)" <gi...@apache.org>.
saadtajwar commented on issue #26099:
URL: https://github.com/apache/datafusion/issues/26099#issuecomment-6474741379
Hey @mohitgurav20 , just wanted to see if you were still working on this one - if not then I'm happy to take a stab, or can review your PR if you are! Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Account for retained sort-key capacity [datafusion]
Posted by "mohitgurav20 (via GitHub)" <gi...@apache.org>.
mohitgurav20 commented on issue #26099:
URL: https://github.com/apache/datafusion/issues/26099#issuecomment-6475174506
@saadtajwar i'm working on it you can feel free to review it later .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org