You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2024/03/27 09:00:43 UTC

[PR] GH-38010: [Python] Construct pyarrow.Field from Arrow-compatible schema object (PyCapsule protocol) [arrow]

jorisvandenbossche opened a new pull request, #40818:
URL: https://github.com/apache/arrow/pull/40818

   ### Rationale for this change
   
   See https://github.com/apache/arrow/issues/38010#issuecomment-2010601912 for more context. Right now for _consuming_ ArrowSchema-compatible objects that implement the PyCapsule interface, we only have the private `_import_from_c_capsule` (on Schema, Field, DataType) and we check for the protocol in the public `pa.schema(..)`.
   
   But that means you currently can only consume objects that represent the schema of a batch (struct type), and not schemas of individual arrays. 
   
   ### What changes are included in this PR?
   
   Expand the `pa.field(..)` constructor to accept objects implementing the protocol method.
   
   ### Are these changes tested?
   
   TODO
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2057761919

   After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit b842b530d14a752697f4283c33f16f2f293713ff.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/23847508946) has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field from Arrow-compatible schema object (PyCapsule protocol) [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2022252483

   :warning: GitHub issue #38010 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555397158


##########
docs/source/python/extending_types.rst:
##########
@@ -53,6 +53,31 @@ support for this protocol by checking for the presence of those methods, and
 therefore accept any Arrow data (instead of harcoding support for a specific
 Arrow producer such as PyArrow).
 
+For consuming data through this protocol with PyArrow, the following constructors
+can be used to create the various PyArrow objects:
+
++----------------------------+-----------------------------------------------+--------------------+
+| Result class               | Mapped Arrow type                             | Supported protocol |

Review Comment:
   Whoops, that's a left-over from the original table I copied from elsewhere for the table layout ;) 
   Changed to "PyArrow constructor"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #40818:
URL: https://github.com/apache/arrow/pull/40818


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field from Arrow-compatible schema object (PyCapsule protocol) [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2022258504

   cc @kylebarron @pitrou @wjones127
   
   I don't particularly like the fact that doesn't really fit into the existing `pa.field(..)` signature, having to make other keywords optional / obsolete in case of passing a ArrowSchemaExportable object (also the `nullable` keyword would currently be silently ignored if you would specif that in this case) 
   
   But the only alternative I can think of is adding something like a `pa.Field.from_arrow(..)` constructor method


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555441823


##########
python/pyarrow/types.pxi:
##########
@@ -3504,6 +3504,14 @@ def field(name, type, bint nullable=True, metadata=None):
     >>> pa.struct([field])
     StructType(struct<key: int32>)
     """
+    if hasattr(name, "__arrow_c_schema__"):
+        if not (type is None and metadata is None):
+            raise ValueError(

Review Comment:
   Yeah, it's true that TypeError might be a better fit given that the passed value is a different type than the expected one, but in general in other similar cases in pyarrow (eg the `table(..)` or `array(..)` constructors) when we have such parameters that can not be specified in certain circumstances, it seems we are currently always using ValueError for that.
   
   (in this case it's also raising just when it has _some value_ (that is not non-None), i.e. it's not checking for a specific type)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555453378


##########
python/pyarrow/table.pxi:
##########
@@ -1344,17 +1344,28 @@ cdef class ChunkedArray(_PandasConvertible):
             A capsule containing a C ArrowArrayStream struct.
         """
         cdef:
+            ChunkedArray chunked
             ArrowArrayStream* c_stream = NULL
 
         if requested_schema is not None:
-            out_type = DataType._import_from_c_capsule(requested_schema)
-            if self.type != out_type:
-                raise NotImplementedError("Casting to requested_schema")
+            target_type = DataType._import_from_c_capsule(requested_schema)
+
+            if target_type != self.type:
+                try:
+                    chunked = self.cast(target_type, safe=True)

Review Comment:
   Opened https://github.com/apache/arrow/issues/41064



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1540723914


##########
python/pyarrow/types.pxi:
##########
@@ -3504,6 +3504,14 @@ def field(name, type, bint nullable=True, metadata=None):
     >>> pa.struct([field])
     StructType(struct<key: int32>)
     """
+    if hasattr(name, "__arrow_c_schema__"):
+        if not (type is None and metadata is None):
+            raise ValueError(

Review Comment:
   Make this `TypeError`?



##########
python/pyarrow/table.pxi:
##########
@@ -1437,6 +1451,21 @@ def chunked_array(arrays, type=None):
 
     if isinstance(arrays, Array):
         arrays = [arrays]
+    elif hasattr(arrays, "__arrow_c_stream__"):
+        if type is not None:
+            requested_type = type.__arrow_c_schema__()
+        else:
+            requested_type = None
+        capsule = arrays.__arrow_c_stream__(requested_type)
+        result = ChunkedArray._import_from_c_capsule(capsule)
+        if type is not None and result.type != type:

Review Comment:
   Why not use `requested_type` here?



##########
docs/source/python/extending_types.rst:
##########
@@ -37,14 +37,14 @@ under the hood, you can implement the following methods on those objects:
 
 - ``__arrow_c_schema__`` for schema or type-like objects.
 - ``__arrow_c_array__`` for arrays and record batches (contiguous tables).
-- ``__arrow_c_stream__`` for chunked tables or streams of data.
+- ``__arrow_c_stream__`` for chunked arrays or tables, or streams of data.

Review Comment:
   ```suggestion
   - ``__arrow_c_stream__`` for chunked arrays, tables and streams of data.
   ```



##########
python/pyarrow/tests/test_table.py:
##########
@@ -493,6 +493,47 @@ def test_recordbatch_dunder_init():
         pa.RecordBatch()
 
 
+def test_chunked_array_c_array_interface():
+    class ArrayWrapper:
+        def __init__(self, array):
+            self.array = array
+
+        def __arrow_c_array__(self, requested_schema=None):
+            return self.array.__arrow_c_array__(requested_schema)
+
+    data = pa.array([1, 2, 3], pa.int64())
+    chunked = pa.chunked_array([data])
+    wrapper = ArrayWrapper(data)
+
+    # Can roundtrip through the wrapper.
+    result = pa.chunked_array(wrapper)
+    assert result == chunked
+
+    # Can also import with a type that implementer can cast to.
+    result = pa.chunked_array(wrapper, type=pa.int16())

Review Comment:
   Also test with a `TypeWrapper` that would wrap the `pa.int16` field and only expose `__arrow_c_schema__`?



##########
python/pyarrow/tests/test_cffi.py:
##########
@@ -692,8 +692,14 @@ def test_roundtrip_chunked_array_capsule_requested_schema():
     imported_chunked = pa.ChunkedArray._import_from_c_capsule(capsule)
     assert imported_chunked == chunked
 
-    # Casting to something else should error
+    # Casting to something else should error if not possible
     requested_type = pa.binary()
     requested_capsule = requested_type.__arrow_c_schema__()
-    with pytest.raises(NotImplementedError):
+    capsule = chunked.__arrow_c_stream__(requested_capsule)
+    imported_chunked = pa.ChunkedArray._import_from_c_capsule(capsule)
+    assert imported_chunked == chunked.cast(pa.binary())
+
+    requested_type = pa.int64()
+    requested_capsule = requested_type.__arrow_c_schema__()
+    with pytest.raises(ValueError):

Review Comment:
   Check the error message as well?



##########
python/pyarrow/tests/test_types.py:
##########
@@ -1335,3 +1335,25 @@ def __arrow_c_schema__(self):
     wrapped_schema = Wrapper(schema)
 
     assert pa.schema(wrapped_schema) == schema
+
+
+def test_field_import_c_schema_interface():
+    class Wrapper:
+        def __init__(self, field):
+            self.field = field
+
+        def __arrow_c_schema__(self):
+            return self.field.__arrow_c_schema__()
+
+    field = pa.field("field_name", pa.int32(), metadata={"key": "value"})
+    wrapped_field = Wrapper(field)
+
+    assert pa.field(wrapped_field) == field
+
+    with pytest.raises(ValueError, match="cannot specify 'type'"):

Review Comment:
   Should be `TypeError`?



##########
docs/source/python/extending_types.rst:
##########
@@ -53,6 +53,31 @@ support for this protocol by checking for the presence of those methods, and
 therefore accept any Arrow data (instead of harcoding support for a specific
 Arrow producer such as PyArrow).
 
+For consuming data through this protocol with PyArrow, the following constructors
+can be used to create the various PyArrow objects:
+
++----------------------------+-----------------------------------------------+--------------------+
+| Result class               | Mapped Arrow type                             | Supported protocol |

Review Comment:
   "Mapped Arrow type" as a column heading is a bit strange as the listed APIs are not types at all. 



##########
python/pyarrow/table.pxi:
##########
@@ -1344,17 +1344,28 @@ cdef class ChunkedArray(_PandasConvertible):
             A capsule containing a C ArrowArrayStream struct.
         """
         cdef:
+            ChunkedArray chunked
             ArrowArrayStream* c_stream = NULL
 
         if requested_schema is not None:
-            out_type = DataType._import_from_c_capsule(requested_schema)
-            if self.type != out_type:
-                raise NotImplementedError("Casting to requested_schema")
+            target_type = DataType._import_from_c_capsule(requested_schema)
+
+            if target_type != self.type:
+                try:
+                    chunked = self.cast(target_type, safe=True)

Review Comment:
   We would ideally like to cast one chunk at a time when the stream is being read, to reduce memory consumption. Perhaps open a GH issue for it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2026901135

   OK, updated this PR with the suggestion for the `pa.field()` API, and further also added support for the protocol in `pa.chunked_array()` (we recently added support to ChunkedArray for the C interface, but also here didn't yet have a public API to consume this).
   
   With that, I think we have a public consume function or method for all cases (DataType is covered through `pa.field()`), and I also added a table to the documentation listing all the different functions that can be used to create pyarrow objects using the protocol.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555404445


##########
python/pyarrow/table.pxi:
##########
@@ -1344,17 +1344,28 @@ cdef class ChunkedArray(_PandasConvertible):
             A capsule containing a C ArrowArrayStream struct.
         """
         cdef:
+            ChunkedArray chunked
             ArrowArrayStream* c_stream = NULL
 
         if requested_schema is not None:
-            out_type = DataType._import_from_c_capsule(requested_schema)
-            if self.type != out_type:
-                raise NotImplementedError("Casting to requested_schema")
+            target_type = DataType._import_from_c_capsule(requested_schema)
+
+            if target_type != self.type:
+                try:
+                    chunked = self.cast(target_type, safe=True)

Review Comment:
   @paleolimbot added this on-the-fly casting for RecordBatchReader in https://github.com/apache/arrow/pull/40070, and it seems that for Table's stream export method, we convert this to a RecordBatchReader and that way Table also already supports casting chunk by chunk. 
   We could indeed do something similar for ChunkedArray, will open an issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2042199016

   @pitrou thanks for the review!
   
   > Also test with a `TypeWrapper` that would wrap the `pa.int16` field and only expose `__arrow_c_schema__`?
   
   In hindsight, we actually don't yet support this in any of our constructors (also `pa.array(..)` and `pa.table(..)` require an actual pyarrow type or schema). So reverted the test changes I did here and will leave this as a follow-up to tackle generally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field from Arrow-compatible schema object (PyCapsule protocol) [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #40818:
URL: https://github.com/apache/arrow/pull/40818#issuecomment-2022281640

   I think this could be slightly smarter:
   1. disallow `type`
   2. if `nullable` or `metadata` is given, update the resulting field with the given metadata
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555408483


##########
python/pyarrow/table.pxi:
##########
@@ -1437,6 +1451,21 @@ def chunked_array(arrays, type=None):
 
     if isinstance(arrays, Array):
         arrays = [arrays]
+    elif hasattr(arrays, "__arrow_c_stream__"):
+        if type is not None:
+            requested_type = type.__arrow_c_schema__()
+        else:
+            requested_type = None
+        capsule = arrays.__arrow_c_stream__(requested_type)
+        result = ChunkedArray._import_from_c_capsule(capsule)
+        if type is not None and result.type != type:

Review Comment:
   If `type` is None then `requested_type` is None, so it shouldn't matter which of the two I am checking here. But since this if clause is to check for whether we still need to cast to `type` (and not `requested_type`, which is a capsule not a pyarrow data type), it seems more logical to check for `type is not None`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555795436


##########
docs/source/python/extending_types.rst:
##########
@@ -53,6 +53,31 @@ support for this protocol by checking for the presence of those methods, and
 therefore accept any Arrow data (instead of harcoding support for a specific
 Arrow producer such as PyArrow).
 
+For consuming data through this protocol with PyArrow, the following constructors
+can be used to create the various PyArrow objects:
+
++----------------------------+-----------------------------------------------+--------------------+
+| Result class               | PyArrow constructor                           | Supported protocol |
++============================+===============================================+====================+
+| :class:`Array`             | :func:`pyarrow.array`                         | array              |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`ChunkedArray`      | :func:`pyarrow.chunked_array`                 | array, stream      |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`RecordBatch`       | :func:`pyarrow.record_batch`                  | array              |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Table`             | :func:`pyarrow.table`                         | array, stream      |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`RecordBatchReader` | :meth:`pyarrow.RecordBatchReader.from_stream` | stream             |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Field`             | :func:`pyarrow.record_batch`                  | schema             |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Schema`            | :func:`pyarrow.record_batch`                  | schema             |

Review Comment:
   Yes of course, thanks for the catch!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-38010: [Python] Construct pyarrow.Field and ChunkedArray through Arrow PyCapsule Protocol [arrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #40818:
URL: https://github.com/apache/arrow/pull/40818#discussion_r1555702181


##########
docs/source/python/extending_types.rst:
##########
@@ -53,6 +53,31 @@ support for this protocol by checking for the presence of those methods, and
 therefore accept any Arrow data (instead of harcoding support for a specific
 Arrow producer such as PyArrow).
 
+For consuming data through this protocol with PyArrow, the following constructors
+can be used to create the various PyArrow objects:
+
++----------------------------+-----------------------------------------------+--------------------+
+| Result class               | PyArrow constructor                           | Supported protocol |
++============================+===============================================+====================+
+| :class:`Array`             | :func:`pyarrow.array`                         | array              |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`ChunkedArray`      | :func:`pyarrow.chunked_array`                 | array, stream      |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`RecordBatch`       | :func:`pyarrow.record_batch`                  | array              |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Table`             | :func:`pyarrow.table`                         | array, stream      |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`RecordBatchReader` | :meth:`pyarrow.RecordBatchReader.from_stream` | stream             |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Field`             | :func:`pyarrow.record_batch`                  | schema             |
++----------------------------+-----------------------------------------------+--------------------+
+| :class:`Schema`            | :func:`pyarrow.record_batch`                  | schema             |

Review Comment:
   Shoud this be `pyarrow.field()` and `pyarrow.schema()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org