[jira] [Created] (ARROW-2041) [Python] pyarrow.serialize has high overhead for list of NumPy arrays

["Richard Shin (JIRA)" <ji...@apache.org>]

Richard Shin created ARROW-2041:
-----------------------------------

             Summary: [Python] pyarrow.serialize has high overhead for list of NumPy arrays
                 Key: ARROW-2041
                 URL: https://issues.apache.org/jira/browse/ARROW-2041
             Project: Apache Arrow
          Issue Type: Bug
            Reporter: Richard Shin


{{Python 2.7.12 (default, Nov 20 2017, 18:23:56)}}
{{[GCC 5.4.0 20160609] on linux2}}
{{Type "help", "copyright", "credits" or "license" for more information.}}
{{>>> import pyarrow as pa, numpy as np}}
{{>>> arrays = [np.arange(100, dtype=np.int32) for _ in range(10000)]}}
{{>>> with open('test.pyarrow', 'w') as f:}}
{{... f.write(pa.serialize(arrays).to_buffer().to_pybytes())}}
{{...}}
{{>>> import cPickle as pickle}}
{{>>> pickle.dump(arrays, open('test.pkl', 'w'), pickle.HIGHEST_PROTOCOL)}}

test.pyarrow is 6.2 MB, while test.pkl is only 4.2 MB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)