You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2024/09/27 07:42:50 UTC
[PR] Align buffers from Python (FFI) [arrow-rs]
EnricoMi opened a new pull request, #6472:
URL: https://github.com/apache/arrow-rs/pull/6472
# Which issue does this PR close?
Together with #6462 fixes #6471.
# Rationale for this change
Data coming from Python could be misaligned. Using them will fail (see #6471):
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type.
Before importing buffer through FFI, please make sure the allocation is aligned.
Costly copying the data is better than failing and not being able to use the data.
# What changes are included in this PR?
Call existing `ArrayData.align_buffers` right after array data are retrieved via FFI (as suggested by the error message).
# Are there any user-facing changes?
Data become useable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align buffers from Python (FFI) [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6472:
URL: https://github.com/apache/arrow-rs/pull/6472#issuecomment-2378636821
I think there is an expectation that FFI doesn't copy buffers, and that data provided is aligned as per the arrow specification.
This is actually a bug in pyarrow and there are some solutions on the linked issue - https://github.com/apache/arrow/issues/32276#issuecomment-1514695089
That being said it might be good to document this further as this bug has been around for a while now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align buffers from Python (FFI) [arrow-rs]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #6472:
URL: https://github.com/apache/arrow-rs/pull/6472#issuecomment-2379234069
@tustvold How does Rust Arrow Flight avoid mis-aligned buffers given it also uses gRPC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align buffers from Python (FFI) [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6472:
URL: https://github.com/apache/arrow-rs/pull/6472#issuecomment-2379304160
Tonic, our gRPC implementation, provides a zero-copy decode mechanism using Bytes, this internally works by taking refcounted slices of the underlying message payload. This is then fed into the IPC decoder which then constructs buffers from this, copying when necessary. This does mean that potentially some data is copied twice, once from the network socket and once into the destination buffer, but in practice some copying is unavoidable with a networked protocol and practically the network throughput will be the major bottleneck regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org