[jira] [Closed] (ARROW-2041) [Python] pyarrow.serialize has high overhead for list of NumPy arrays

["Wes McKinney (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney closed ARROW-2041.
-------------------------------
    Resolution: Won't Fix

I looked at this. The breakdown of parts of each NumPy arrays is as follows:

* 400 bytes of data
* 176 bytes of metadata
* 64 bytes of padding

So there's a few things that can be done, and you're free to open some JIRA issues:

* Create more compact metadata for Tensors -- Protocol Buffers could be a good option that's smaller than that Flatbuffers table that's produced. 
* Reduce the padding requirement to 8 bytes instead of 64 bytes. 

> [Python] pyarrow.serialize has high overhead for list of NumPy arrays
> ---------------------------------------------------------------------
>
>                 Key: ARROW-2041
>                 URL: https://issues.apache.org/jira/browse/ARROW-2041
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Richard Shin
>            Priority: Minor
>              Labels: Performance
>             Fix For: 0.15.0
>
>
> {{Python 2.7.12 (default, Nov 20 2017, 18:23:56)}}
> {{[GCC 5.4.0 20160609] on linux2}}
> {{Type "help", "copyright", "credits" or "license" for more information.}}
> {{>>> import pyarrow as pa, numpy as np}}
> {{>>> arrays = [np.arange(100, dtype=np.int32) for _ in range(10000)]}}
> {{>>> with open('test.pyarrow', 'w') as f:}}
> {{... f.write(pa.serialize(arrays).to_buffer().to_pybytes())}}
> {{...}}
> {{>>> import cPickle as pickle}}
> {{>>> pickle.dump(arrays, open('test.pkl', 'w'), pickle.HIGHEST_PROTOCOL)}}
> test.pyarrow is 6.2 MB, while test.pkl is only 4.2 MB.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)