Regarding pyarrow package size for aws lambda

[thanuj d <dt...@gmail.com>]

Hello,
      I am Thanuj, I am trying to use pyarrow in aws lambda. The problem I
am facing is with the size more than 250mb. I tried several options but not
successful I tried removing Gandiva files from package but it gives me
corrupt file error. Can you please help me with this on how to disable
gandiva and other options.
-- 
*Thanks*

*D Thanuj,*
*dthanuj28@gmail.com <dt...@gmail.com>.*

Re: Regarding pyarrow package size for aws lambda

[Wes McKinney <we...@gmail.com>]

hi Thanuj,

It is straightforward to alter flags in

https://github.com/apache/arrow/blob/master/python/manylinux1/build_arrow.sh

and create a slimmed down wheel. See the README for instructions how
to run the script (It uses docker-compose so should be easily
reproducible on any machine).

It's possible that we could publish a pyarrow-slim wheel on PyPI in
future releases, but we would need help from volunteers to implement
and maintain this. My organization is not being funded to support
wheels and we have already spent too much time on wheel-related issues
to take on a project like this.

Note that we also have an outstanding problem of shared libraries
being duplicated in the Linux wheels so that is surely exacerbating
this problem. We tried to fix it but the fix caused more problems so
we (at least, my colleagues and I) gave up until someone else could
spend time looking at it.

Thanks

On Fri, Dec 20, 2019 at 4:36 PM thanuj d <dt...@gmail.com> wrote:
>
> Hello,
>       I am Thanuj, I am trying to use pyarrow in aws lambda. The problem I
> am facing is with the size more than 250mb. I tried several options but not
> successful I tried removing Gandiva files from package but it gives me
> corrupt file error. Can you please help me with this on how to disable
> gandiva and other options.
> --
> *Thanks*
>
> *D Thanuj,*
> *dthanuj28@gmail.com <dt...@gmail.com>.*