[jira] [Comment Edited] (ARROW-6776) [Python] Need a lite version of pyarrow

["Antoine Pitrou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147676#comment-17147676 ] 

Antoine Pitrou edited comment on ARROW-6776 at 6/29/20, 10:47 AM:
------------------------------------------------------------------

The latest PyArrow wheels are much lighter:
{code}
$ du -hs venv-3.7/lib/python3.7/site-packages/pyarrow/
57M	venv-3.7/lib/python3.7/site-packages/pyarrow/
{code}

PS: see here for nightly PyArrow wheels:
https://arrow.apache.org/docs/python/install.html#installing-nightly-packages



was (Author: pitrou):
The latest PyArrow wheels (*) are much lighter:
{code}
$ du -hs venv-3.7/lib/python3.7/site-packages/pyarrow/
57M	venv-3.7/lib/python3.7/site-packages/pyarrow/
{code}

PS: see here for nightly PyArrow wheels:
https://arrow.apache.org/docs/python/install.html#installing-nightly-packages


> [Python] Need a lite version of pyarrow
> ---------------------------------------
>
>                 Key: ARROW-6776
>                 URL: https://issues.apache.org/jira/browse/ARROW-6776
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 0.14.1
>            Reporter: Haowei Yu
>            Priority: Major
>
> Currently I am building a library packages on top of pyarrow, so I include pyarrow as a dependency and ship it to our customer. However, when our customer installed our packages, it will also install pyarrow and pyarrow's dependency (numpy). However the dependency size is huge. 
> {code:bash}
> (py36env) [hyu@c6x64-hyu-newuser-final-clone connector]$ ls -l --block-size=M /home/hyu/py36env/lib/python3.6/site-packages/pyarrow/ 
> total 186M
> {code}
>  And numpy is around 80MB. Total is more than 250 MB.
> Our customer want to bundle all dependency and run the code inside AWS Lambda, however they hit the size limit and failed to run the code.
> Looking into the pyarrow, I saw multiple .so files are shipped both with and without version suffix, I wonder if you can remove the one of them (either with or without suffix), it will at least reduce the package size by half.
> Further, our library just want to use IPC and read data as record batch, I don't need arrow flight at all (which is the biggest .so file and takes around 100 MB). I wonder if you can push a lite version of the pyarrow so that I can specify lite version as the dependency. Or maybe I need to build my own lite version and push it pypi. However, this approach cause further problem if our customer is using the "fat" version of pyarrow unless you the change the namespace of lite version of pyarrow.
> Another alternative is that I bundle the pyarrow with our library ( copy the whole directory into vendored namespace) and ship it to our customer without specifying pyarrow as a dependency. The advantage of this one is that I can build pyarrow with whatever option/sub-module/libraries I need. However, I tried a lot but failed because pyarrow use absolute import and it will fail to import the script in the new location. 
> Any insight how I should resolve this issue?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)