You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2025/02/17 00:55:45 UTC
[I] mallocator: align allocations to 64 bytes [arrow-go]
lidavidm opened a new issue, #282:
URL: https://github.com/apache/arrow-go/issues/282
### Describe the enhancement requested
Some types want 16 byte (or more) alignment, such as decimal256, and it's easiest just to overallocate + align to avoid issues (even if it's not strictly _required_ by the spec, many things in practice assume it). We have to overallocate since the allocator API is not given an alignment.
Context: https://github.com/apache/arrow-adbc/issues/2526
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661910996
Ah, but Go directly uses `byte[]`, so we have nowhere to stash the "real" pointer...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661973784
I was looking at that. In that case maybe we just make `Buffer` handle the alignment and leave the allocator out of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661925478
The best thing I can think of is to wrap [aligned_alloc](https://en.cppreference.com/w/c/memory/aligned_alloc) but that carries a number of caveats (requires C11, not portable to Windows, does not support realloc, cannot align an allocation smaller than the alignment). Maybe we can consider this if there's enough demand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #282: mallocator: align allocations to 64 bytes
URL: https://github.com/apache/arrow-go/issues/282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661987816
The other complication is resizing, since resizing won't preserve the alignment; we may have to copy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2662087768
Well then we'd have to allocate and try again?
I'm just saying that, if the allocator already does the padding trick underneath (or otherwise somehow supports aligning the allocation), then we shouldn't add the padding ourselves in Buffer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661926558
We're still using a `memory. Buffer` and can use slices, so it wouldn't be difficult to modify the buffer struct to stash the "real" pointer. We stash the pointer and then have the slice start at the aligned data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661989368
Resizing frequently requires a copy anyways, so as long as we handle it properly it should be okay I think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661979133
That's a good point! It's definitely easier to manage in the buffer class and use the slice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661978126
Well otherwise we'd have to change the allocator API to return the `(base, aligned)` pair
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2662001904
Yeah.
The other thing is the native Go allocator already does the trick, so we need to take care not to double-do it. So I'm thinking I'll either add a new "AllocateAligned" that does return the pair, or a flag on the allocator interface to indicate whether the allocator handles it natively or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2661976912
Possibly, we'd have to make sure that the release callback still releases the original pointer, but otherwise it could work if we make sure to overallocate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] mallocator: align allocations to 64 bytes [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #282:
URL: https://github.com/apache/arrow-go/issues/282#issuecomment-2662004672
Go has an `AlignOf` method that might help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org