You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/02/05 15:17:10 UTC
[I] Preallocate single `Vec` in `get_ranges` for LocalFilesystem [arrow-rs-object-store]
Dandandan opened a new issue, #634:
URL: https://github.com/apache/arrow-rs-object-store/issues/634
I wondered if it makes sense to do the allocation in bulk in `get_ranges` instead of doing it per range - it seems that is beneficial for locality and saves some allocations.
_Originally posted by @Dandandan in https://github.com/apache/arrow-rs-object-store/pull/628#discussion_r2769689847_
This also helps with locality when decoding the data.
The downside is that a user won't be able to deallocate certain parts of the data (without explicitly reallocating or copying).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Preallocate single `Vec` in `get_ranges` for LocalFilesystem [arrow-rs-object-store]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #634:
URL: https://github.com/apache/arrow-rs-object-store/issues/634#issuecomment-3938735763
Probably you're right - let's close it for now and see if it pops up in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Preallocate single `Vec` in `get_ranges` for LocalFilesystem [arrow-rs-object-store]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #634:
URL: https://github.com/apache/arrow-rs-object-store/issues/634#issuecomment-3854355271
@AdamGS since you are also interested in this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Preallocate single `Vec` in `get_ranges` for LocalFilesystem [arrow-rs-object-store]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #634:
URL: https://github.com/apache/arrow-rs-object-store/issues/634#issuecomment-3926729485
I think there is probably a threshold where it makes sense to just let your memory allocator do its job, I can see an advantage if people are requesting 5 byte ranges (which is silly but anyway), but if you're requesting 5KB ranges, forcing the memory allocator to find a 50KB contiguous region, when 10 separate 5KB regions would suffice seems counter-productive. Normally memory is much more precious than CPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Preallocate single `Vec` in `get_ranges` for LocalFilesystem [arrow-rs-object-store]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed issue #634: Preallocate single `Vec` in `get_ranges` for LocalFilesystem
URL: https://github.com/apache/arrow-rs-object-store/issues/634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org