[jira] [Commented] (ARROW-2295) Add to_numpy functions

["Jim Pivarski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394196#comment-16394196 ] 

Jim Pivarski commented on ARROW-2295:
-------------------------------------

I second this and would like to request that the Numpy interface has more low-level access to Arrow structures. For instance, ListArray is internally represented as two arrays: offsets and contents, and there are applications where we'd want to get a zero-copy view of these arrays. The to_pandas() function constructs a Numpy object array of subarrays, which is a performance bottleneck if you really do want the original offsets and contents.

This function could be an inverse of pyarrow.ListArray.from_arrays, something that returns the offsets and contents as Numpy arrays for a List<Primitive> and something more complex for general cases (a dict from strings representing a place in the hierarchy to Numpy arrays?).

A simpler interface that could be implemented immediately would be one that returns the raw bytes of the Arrow buffer, to let us identify its contents using [the Arrow spec|[https://github.com/apache/arrow/blob/master/format/Layout.md].] But that doesn't make use of the dtype (probably just set it to uint8) and would probably make more sense as a raw __buffer__. (Should that be a separate JIRA ticket?)

 

> Add to_numpy functions
> ----------------------
>
>                 Key: ARROW-2295
>                 URL: https://issues.apache.org/jira/browse/ARROW-2295
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: Lawrence Chan
>            Priority: Minor
>
> There are `to_pandas()` functions, but no `to_numpy()` functions. I'd like to propose that we include both.
> Also, `pyarrow.lib.Array.to_pandas()` returns a `numpy.ndarray`, which imho is very confusing :). I think it would be more intuitive for the `to_pandas()` functions to return `pandas.Series` and `pandas.DataFrame` objects, and the `to_numpy()` functions to return `numpy.ndarray` and either a ordered dict of `numpy.ndarray` or a structured `numpy.ndarray` depending on a flag, for example. The `to_pandas()` function is of course welcome to use the `to_numpy()` func to avoid the additional index and whatnot of the `pandas.Series`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)