You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ding-young (via GitHub)" <gi...@apache.org> on 2025/08/26 13:12:36 UTC
[I] `get_slice_memory_size` underestimates memory for `StringViewArray` [arrow-rs]
ding-young opened a new issue, #8230:
URL: https://github.com/apache/arrow-rs/issues/8230
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
It is found (in https://github.com/apache/datafusion/pull/17029) that `get_sliced_memory_size` underestimated the memory size for `StringViewArray`. The diff was quite large i.e. when correct batch size was `1392730` bytes, it returned only `163840` bytes.
This is because payload buffers are not taken account of although these payload buffers are simply `cloned()` when we `slice()` `StringViewArray`
See
https://github.com/apache/arrow-rs/blob/a620957bc98b7aa14faec10635bb798932f00bf9/arrow-data/src/data.rs#L464-L476
https://github.com/apache/arrow-rs/blob/a620957bc98b7aa14faec10635bb798932f00bf9/arrow-data/src/data.rs#L1743-L1752
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
`get_sliced_memory_size` should include the capacity for payload buffers.
If this fix isn’t ideal, we should at least document which buffers are being counted in the calculation.
**Additional context**
<!--
Add any other context about the problem here.
-->
https://github.com/apache/datafusion/pull/17315
here's related pr for temporal fix in datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_slice_memory_size` underestimates memory for `StringViewArray` [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #8230:
URL: https://github.com/apache/arrow-rs/issues/8230#issuecomment-5111738964
some pointers
https://github.com/apache/arrow-rs/blob/12091af35e5070dae33e86fbe988d9319521cd16/arrow-data/src/data.rs#L514-L589
could likely add to the bottom, checking if we are a utf8/binary view type, and just take the sizes of the values buffers (could be multiple) unchanged since if we slice them we aren't gonna slice the values buffers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_slice_memory_size` underestimates memory for `StringViewArray` [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #8230: `get_slice_memory_size` underestimates memory for `StringViewArray`
URL: https://github.com/apache/arrow-rs/issues/8230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `get_slice_memory_size` underestimates memory for `StringViewArray` [arrow-rs]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #8230:
URL: https://github.com/apache/arrow-rs/issues/8230#issuecomment-3224128556
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org