[GitHub] [arrow] spenczar opened a new issue, #34985: Python extension types aren't usable in struct arrays

["spenczar (via GitHub)" <gi...@apache.org>]

spenczar opened a new issue, #34985:
URL: https://github.com/apache/arrow/issues/34985

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I have a custom PyExtensionType. I would like to use that value for a field of an array, and use that array inside a larger data structure.
   
   I'm able to use the extension type directly in `pa.array`, but if I use a `pa.struct` to wrap it up, it fails.
   
   Here is a minimal reproducer. Running `make_array_ok()` does not error. Running `make_struct_array_not_ok()` results in an error.
   
   ```python
   import pyarrow as pa
   
   
   class CustomExtensionType(pa.PyExtensionType):
       def __init__(self):
           pa.PyExtensionType.__init__(self, pa.int64())
   
       def __reduce__(self):
           return CustomExtensionType, ()
   
   
   def make_array_ok():
       data = [1, 2, 3, 4, 5]
       typ = CustomExtensionType()
       return pa.array(data, typ)
   
   
   def make_struct_array_not_ok():
       data = [{"val": 1}, {"val": 2}, {"val": 3}, {"val": 4}, {"val": 5}]
       typ = CustomExtensionType()
       struct_type = pa.struct([("val", typ)])
       return pa.array(data, type=struct_type)
   ```
   
   The error is not very clear to me:
   ```
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/redacted/struct_demo.py", line 22, in make_struct_array_not_working
       return pa.array(data, type=struct_typ)
     File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
     File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
     File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: extension
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] spenczar commented on issue #34985: Python extension types aren't usable in struct arrays

["spenczar (via GitHub)" <gi...@apache.org>]

spenczar commented on issue #34985:
URL: https://github.com/apache/arrow/issues/34985#issuecomment-1500933703

   Actually, making the array explicitly with StructArray.from_arrays works:
   ```
   def make_struct_array_explicitly():
       data = [1, 2, 3, 4, 5]
       typ = CustomExtensionType()
       return pa.StructArray.from_arrays([pa.array(data, typ)], fields=[("val", typ)])
   ```
   
   So I think this is really about `pa.array`'s handling of structs that contain extension types.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org