You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2023/02/03 15:19:15 UTC
[GitHub] [arrow] jorisvandenbossche opened a new issue, #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
jorisvandenbossche opened a new issue, #34031:
URL: https://github.com/apache/arrow/issues/34031
### Describe the enhancement requested
Currently we have the various `_export_to_c` / `_import_from_c` methods for working with the Arrow C Interface that expect integers as arguments for the struct pointers. We could also use `PyCapsule` objects for this instead of integers (or (certainly initially) in addition to), inspired by a similar interface from DLPack.
[DLPack](https://dmlc.github.io/dlpack/latest/index.html) provides a stable in-memory data structure that allows exchanging array data between frameworks. It essentially plays the same role for ndarrays (tensors) as what the Arrow C interface does for arrow-compatible data (columnar data). It also defines a stable C ABI with a similar C struct definitions ([header file](https://github.com/dmlc/dlpack/blob/ca4d00ad3e2e0f410eeab3264d21b8a39397f362/include/dlpack/dlpack.h#L187-L233)).
In the DLPack project, apart from the stable C ABI struct, they also defined a python specification (including a method name to access the protocol, i.e. `__dlpack__`), see https://dmlc.github.io/dlpack/latest/python_spec.html#implementation for the details. And for that specification, they return not a raw pointer, but use a `PyCapsule` object (a python object that represents an "opaque value", such as a pointer, and that can used by C extensions to pass such values through Python code to other C code, https://docs.python.org/3/c-api/capsule.html).
Some details based on their implementation:
- The PyCapsule has a name, and the producer should set that to a well-defined value (giving some protection to not getting a random pointer)
- The consumer renames the capsule (which gives some protection to the C Data pointer being consumed more than once)
- The PyCapsule has a destructor defined that would call the release callback (in case the object never got consumed)
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1416075241
Yes, certainly, if we decide to go this way, we should support it in the places we control that interacts with the C Data Interface like ADBC (we should also check with other produces/consumers if this is possible to use)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1561068632
> > The PyCapsule has a destructor defined that would call the release callback (in case the object never got consumed)
>
> I think this would still be useful.
Yes, we would still keep it, but just using a different check to see when it should be called (checking if the release callback is not null, instead of the changed capsule name, as dlpack does and as I mentioned in the top post)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
URL: https://github.com/apache/arrow/issues/34031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1544103104
> > * The consumer renames the capsule (which gives some protection to the C Data pointer being consumed more than once)
>
> This doesn't seem necessary in our case, as the consumer would typically move the C struct contents.
But can a consumer move the C struct content multiple times (and potentially calling the release callback multiple times?)
The other reason for renaming in the case of DLPack is that it checks for this in the capsule deleter, to only call the release callback of the struct in case it was not consumed/renamed (for example [cupy's implementation](https://github.com/cupy/cupy/blob/cb71365934dd5fa8197578ba55b74370cf07951a/cupy/_core/dlpack.pyx#L79-L85)). But that's in the end the same question I assume, because it also depends on whether it is OK to call the release callback multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1767985228
Hi, I am creating the RC0 right now. I can add this in future RCs if they are created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1561333330
I created a PR on the adbc (https://github.com/apache/arrow-adbc/pull/702) and pyarrow (https://github.com/apache/arrow/pull/35739) side as a small experiment with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1416027637
It would also be nice to do this in the ADBC libraries (which currently use a handwritten Python wrapper for this purpose)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1542306787
> * The consumer renames the capsule (which gives some protection to the C Data pointer being consumed more than once)
This doesn't seem necessary in our case, as the consumer would typically move the C struct contents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1544283137
Indeed, you don't call the release callback when moving the structure:
https://arrow.apache.org/docs/format/CDataInterface.html#moving-an-array
> The consumer can move the ArrowArray structure by bitwise copying or shallow member-wise copying. Then it MUST mark the source structure released (see “released structure” above for how to do it) but without calling the release callback. This ensures that only one live copy of the struct is active at any given time and that lifetime is correctly communicated to the producer.
> As usual, the release callback will be called on the destination structure when it is not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1544107046
Ah, I suppose this is handled by setting the release callback to NULL when moving the C struct content. And then the capsule deleter should check for that (instead of checking for it being renamed) ?
The the guideline for consuming a Arrow C struct through a PyCapsule would be to always move the struct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34031: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34031:
URL: https://github.com/apache/arrow/issues/34031#issuecomment-1548350760
> The PyCapsule has a destructor defined that would call the release callback (in case the object never got consumed)
I think this would still be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org