You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/10/03 16:11:36 UTC
[PR] fix(arrow-select): fix `take` output capacity for sliced lists [arrow-rs]
neilconway opened a new pull request, #11360:
URL: https://github.com/apache/arrow-rs/pull/11360
…le values
`take` on a List, LargeList or Map estimated the output's child capacity as the average row length times the number of rows taken. It worked out the average from the whole child array, including values outside a slice, so taking rows from a small slice of a large array could reserve far more memory than needed, and the result kept that memory. For example, taking 100 rows from a two-row slice of a 1000-row array of 10-element lists used about 4 MB instead of 8 KB.
Work out the average from the child values that the rows use.
# Which issue does this PR close?
- N/A
# Rationale for this change
`take` on a list or map uses the average input row length as part of its estimate for sizing its output buffer. However, it calculated the average row length starting from raw size of the input values buffer, not the visible span. This could result in computing a very large "average row length" for sliced lists and maps, and therefore over-allocating output capacity.
# What changes are included in this PR?
* Use the visible span to compute average row length
* Refactor capacity calculations to avoid duplication
* Add unit test
# Are these changes tested?
Yes; existing tests pass, new test added.
# Are there any significant user-facing changes?
No.
# AI usage
Developed and revised with Claude Code (Opus 5.5), reviewed with Codex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-select): fix `take` output capacity for sliced lists [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #11360:
URL: https://github.com/apache/arrow-rs/pull/11360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-select): fix `take` output capacity for sliced lists [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #11360:
URL: https://github.com/apache/arrow-rs/pull/11360#issuecomment-5975869029
thanks @neilconway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org