You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lllangWV (via GitHub)" <gi...@apache.org> on 2025/01/09 03:17:43 UTC
[I] Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions [arrow]
lllangWV opened a new issue, #45208:
URL: https://github.com/apache/arrow/issues/45208
### Describe the enhancement requested
### Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions
Hello, pyarrow devs!
I have been using the PyArrow extension capability to define custom types, which is extremely useful for extending Arrow's functionality. However, a significant limitation arises when using these custom types with compute functions.
For example, the `FixedShapeTensorType` type, designed as an extension type for `ndarrays`, triggers an error when used with the `pc.equal` function to compare arrays:
#### Example Code
```python
import pyarrow as pa
import pyarrow.compute as pc
tensor_type = pa.fixed_shape_tensor(pa.int32(), (2, 2))
arr_1 = [[1, 2, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]
storage_1 = pa.array(arr_1, pa.list_(pa.int32(), 4))
tensor_array_1 = pa.ExtensionArray.from_storage(tensor_type, storage_1)
arr_2 = [[1, 3, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]
storage_2 = pa.array(arr_2, pa.list_(pa.int32(), 4))
tensor_array_2 = pa.ExtensionArray.from_storage(tensor_type, storage_2)
# This triggers an error
print(pc.equal(tensor_array_1, tensor_array_2))
```
#### Error Message
```bash
return func.call(args, None, memory_pool)
File "pyarrow\\_compute.pyx", line 385, in pyarrow._compute.Function.call
File "pyarrow\\error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\\error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (extension<arrow.fixed_shape_tensor[value_type=int32, shape=[2,2]]>, extension<arrow.fixed_shape_tensor[value_type=int32, shape=[2,2]]>)
```
### Proposed Solution
I believe it would be highly useful for PyArrow to allow users to define custom operator support for extension types, similar to how [Pandas enables operator support for `ExtensionArray`](https://pandas.pydata.org/pandas-docs/stable/development/extending.html#extensionarray-operator-support).
#### Suggested Implementation
Here’s an example for the interface:
```python
class PythonObjectArrowType(pa.ExtensionType):
def __init__(self):
super().__init__(pa.binary(), "parquetdb.PythonObjectArrow")
def __arrow_ext_serialize__(self):
return b""
@classmethod
def __arrow_ext_deserialize__(cls, storage_type, serialized):
return PythonObjectArrowType()
def __arrow_ext_class__(self):
return PythonObjectArrowArray
def to_pandas_dtype(self):
return PythonObjectPandasDtype()
def __arrow_ext_scalar_class__(self):
return PythonObjectArrowScalar
pa.register_extension_type(PythonObjectArrowType())
class PythonObjectArrowScalar(pa.ExtensionScalar):
def as_py(self):
return data_utils.load_python_object(self.value.as_py())
def __eq__(self, other):
return self.value == other.value
class PythonObjectArrowArray(pa.ExtensionArray):
def to_pandas(self, **kwargs):
values = self.storage.to_numpy(zero_copy_only=False)
results = mp_utils.parallel_apply(data_utils.load_python_object, values)
return pd.Series(results)
def to_values(self, **kwargs):
values = self.storage.to_pandas(**kwargs).values
results = mp_utils.parallel_apply(data_utils.load_python_object, values)
return results
```
In this example, the `PythonObjectArrowScalar` class defines an `__eq__` method, enabling custom equality comparisons for the scalar elements. Similarly, the `PythonObjectArrowArray` class can provide custom implementations for data conversion and manipulation.
### Challenges
While defining `__eq__` in the scalar class is straightforward, I am uncertain how this would integrate into compute functions like `pc.equal`. It may require exposing additional hooks or mechanisms in PyArrow to allow users to register their operator implementations.
Please let me know if additional details or examples are needed.
Best,
Logan Lang
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45208:
URL: https://github.com/apache/arrow/issues/45208#issuecomment-2582454469
That's an interesting idea but we probably have to fix nested types first which are currently failing too. More details here:
- https://github.com/apache/arrow/issues/45167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #45208: Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions
URL: https://github.com/apache/arrow/issues/45208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #45208:
URL: https://github.com/apache/arrow/issues/45208#issuecomment-4397390317
In the effort of consolidating issues connected to extension type support in PyArrow and grouping them in one umbrella issue, I am closing this one as a duplicate of https://github.com/apache/arrow/issues/22304
I propose to move the discussion there. Feel free to comment or reopen if you disagree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enhancement Request: Custom Operator Support for PyArrow Extension Types in Compute Functions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #45208:
URL: https://github.com/apache/arrow/issues/45208#issuecomment-2582611856
_I think_ there are two different things mentioned in this issue, one is compute kernels and the support of them for extension types. The other is the Python comparison operators for the Extension arrays.
I am quite sure defining `__eq__` method on Scalar object will not solve the fact that some kernels, `equals` in this example, are not supported for Extension types in C++.
There is an issue opened that covers kernel support for ExtensionTypes and I think it would be worth moving it forward, see https://github.com/apache/arrow/issues/22304. Also connected to the kernels: https://github.com/apache/arrow/issues/33452.
On the other hand it would be worth investigating a bit more, how using Python equality operators could be improved for Extension arrays. Currently we check type equality and value of the storage separately in the tests.
I think the issue connected to this might be: https://github.com/apache/arrow/issues/24348.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org