You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/29 14:19:32 UTC
[I] Reduce overhead to create an Array from ArrayData [arrow-rs]
alamb opened a new issue, #9061:
URL: https://github.com/apache/arrow-rs/issues/9061
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- related to https://github.com/apache/datafusion/pull/19477
While looking at profiles for some of the ClickBench queries, I found that almost 1% of the query time was being spent creating record batches from ArrayData,
<img width="1461" height="994" alt="Screenshot 2025-12-28 at 5 27 02 PM" src="https://github.com/user-attachments/assets/ee95d2a5-d722-4310-aa60-5885d5ae9ad4" />
<img width="1420" height="189" alt="Screenshot 2025-12-28 at 5 29 19 PM" src="https://github.com/user-attachments/assets/278ed5e7-448a-4491-bb6e-26c7510d98f8" />
I looked into it and i think the issue is that even though Arrays get an owned `ArrayData`, they still end up `clone` ing some things. For some arrays this isn't a big deal (a few Arc updates) for some such as StructArray and ByteViewArray it requires an extra allocation
https://github.com/apache/arrow-rs/blob/814ee4227c01fce478bdd3594dd156250286b46e/arrow-array/src/array/struct_array.rs#L348
**Describe the solution you'd like**
I would like to avoid this overhead by directly constructing the arrays from the relevant fields of ArrayData rather than having to clone them
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce overhead to create an Array from ArrayData (`make_array`) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9061: Reduce overhead to create an Array from ArrayData (`make_array`)
URL: https://github.com/apache/arrow-rs/issues/9061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce overhead to create an Array from ArrayData (`make_array`) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9061:
URL: https://github.com/apache/arrow-rs/issues/9061#issuecomment-3779310473
All `impl From<ArrayData>` have been merged. I also double checked that they all now use `ArrayData::into_parts` so this is now complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org