Re: [I] Custom data types in arrow array [arrow]

["llama90 (via GitHub)" <gi...@apache.org>]

llama90 commented on issue #38559:
URL: https://github.com/apache/arrow/issues/38559#issuecomment-1790681605

   Here's what I found.
   
   In the given code, the steps are necessary because `pyarrow` cannot automatically infer the type of the user-defined `Point` objects within the dataset.
   
   When creating an Arrow array, the data type needs to be compatible with Arrow's type system. 
   
   Therefore, the `structured_array` is created to convert the `Point` objects into a format that `pyarrow` can understand, which is a list of dictionaries in this case. 
   
   Then, when calling `pa.array`, the type argument is used to explicitly specify the schema of the Arrow array, ensuring that the data is correctly typed as a structured array with fields `x` and `y` of type int64.
   
   ```python
   import pyarrow as pa
   
   
   class Point:
       def __init__(self, iterable):
           self._data = iterable
   
       @property
       def data(self) -> dict:
           return self._data
   
   
   if __name__ == '__main__':
       pt1 = Point({"x": 1, "y": 2, })
       pt2 = Point({"x": 3, "y": 4, })
       pt3 = Point({"x": 5, "y": 6, })
       pt4 = Point({"x": 7, "y": 8, })
   
       dataset = [pt1, pt2, pt3, pt4]
       structured_array = [pt.data for pt in dataset]
       arrow_array = pa.array(structured_array, type=pa.struct([('x', pa.int64()), ('y', pa.int64())]))
       chunked_array = pa.chunked_array([arrow_array])
       print(chunked_array)
   
   --- output
   
   [
     -- is_valid: all not null
     -- child 0 type: int64
       [
         1,
         3,
         5,
         7
       ]
     -- child 1 type: int64
       [
         2,
         4,
         6,
         8
       ]
   ]
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org