[jira] [Closed] (ARROW-2913) [Python] Exported buffers don't expose type information

["Wes McKinney (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney closed ARROW-2913.
-------------------------------
       Resolution: Won't Fix
    Fix Version/s:     (was: 0.13.0)

> [Python] Exported buffers don't expose type information
> -------------------------------------------------------
>
>                 Key: ARROW-2913
>                 URL: https://issues.apache.org/jira/browse/ARROW-2913
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python
>    Affects Versions: 0.10.0
>            Reporter: Antoine Pitrou
>            Priority: Major
>
> Using the {{buffers()}} method on array gives you a list of buffers backing the array, but those buffers lose typing information:
> {code:python}
> >>> a = pa.array(range(10))
> >>> a.type
> DataType(int64)
> >>> buffers = a.buffers()
> >>> [(memoryview(buf).format, memoryview(buf).shape) for buf in buffers]
> [('b', (2,)), ('b', (80,))]
> {code}
> Conversely, Numpy exposes type information in the Python buffer protocol:
> {code:python}
> >>> a = pa.array(range(10))
> >>> memoryview(a.to_numpy()).format
> 'l'
> >>> memoryview(a.to_numpy()).shape
> (10,)
> {code}
> Exposing type information on buffers could be important for third-party systems, such as Dask/distributed, for type-based data compression when serializing.
> Since our C++ buffers are not typed, it's not obvious how to solve this. Should we return tensors instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)