Arrow on PyPy3 patch

[Niklas B <ni...@enplore.com>]

Hi,

I’ve been (together with the PyPy team) working on getting arrow to build on PyPy3. I’m not looking for full feature capability, but specifically getting it to work with pandas read_parquet/to_parquet which it now does. There were a few roadblocks solved by the awesome Matti Picus on the PyPy Team and we now have a patch that successful builds pyarrow on PyPy3. PyPy3 side has already been patched. 

The patch for pyarrow is on https://gist.githubusercontent.com/mattip/c9c8398b58721ae5893dc8134c353f28/raw/0daff3e11ceed6dcde485a56e6b8bd2b7ca48bbc/gistfile1.txt

A Dockerfile which builds everything is available on https://github.com/bivald/pyarrow-on-pypy3/blob/feature/latest-pypy-latest-pyarrow/Dockerfile (https://github.com/bivald/pyarrow-on-pypy3/tree/feature/latest-pypy-latest-pyarrow)

A surprisingly amount of tests passes (such as all the parquet tests when I tested it last) but some other areas segfaults, but neither of them is super important for me right now.

Would the arrow project be open to a PR with the above patch, even though it doesn’t give you full PyPy support?

Regards,
Niklas

Re: Arrow on PyPy3 patch

[Wes McKinney <we...@gmail.com>]

Either way having a Dockerfile in the project to test with PyPy sounds like
a good idea.

On Thu, Oct 22, 2020 at 6:37 AM Antoine Pitrou <an...@python.org> wrote:

>
> We can, but we cannot be expected to act if something breaks.  So this
> would be wasting CPU resources for little use.
>
> Regards
>
> Antoine.
>
>
> Le 22/10/2020 à 13:33, Krisztián Szűcs a écrit :
> > On Thu, Oct 22, 2020 at 1:21 PM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >>
> >> Hi Niklas,
> >>
> >> Le 22/10/2020 à 13:16, Niklas B a écrit :
> >>>
> >>> Would the arrow project be open to a PR with the above patch, even
> though it doesn’t give you full PyPy support?
> >>
> >> I think that would be ok, but you'll have to provide maintenance when
> >> needed, because I think we're unlikely to make PyPy mandatory in our CI
> >> setup.
> > We can set up a nightly build though, just need to select the tests
> > cases which are expected to pass.
> >>
> >> Regards
> >>
> >> Antoine.
>

Re: Arrow on PyPy3 patch

[Antoine Pitrou <an...@python.org>]

We can, but we cannot be expected to act if something breaks.  So this
would be wasting CPU resources for little use.

Regards

Antoine.


Le 22/10/2020 à 13:33, Krisztián Szűcs a écrit :
> On Thu, Oct 22, 2020 at 1:21 PM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Hi Niklas,
>>
>> Le 22/10/2020 à 13:16, Niklas B a écrit :
>>>
>>> Would the arrow project be open to a PR with the above patch, even though it doesn’t give you full PyPy support?
>>
>> I think that would be ok, but you'll have to provide maintenance when
>> needed, because I think we're unlikely to make PyPy mandatory in our CI
>> setup.
> We can set up a nightly build though, just need to select the tests
> cases which are expected to pass.
>>
>> Regards
>>
>> Antoine.

Re: Arrow on PyPy3 patch

[Krisztián Szűcs <sz...@gmail.com>]

On Thu, Oct 22, 2020 at 1:21 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Niklas,
>
> Le 22/10/2020 à 13:16, Niklas B a écrit :
> >
> > Would the arrow project be open to a PR with the above patch, even though it doesn’t give you full PyPy support?
>
> I think that would be ok, but you'll have to provide maintenance when
> needed, because I think we're unlikely to make PyPy mandatory in our CI
> setup.
We can set up a nightly build though, just need to select the tests
cases which are expected to pass.
>
> Regards
>
> Antoine.

Re: Arrow on PyPy3 patch

[Antoine Pitrou <an...@python.org>]

Hi Niklas,

Le 22/10/2020 à 13:16, Niklas B a écrit :
> 
> Would the arrow project be open to a PR with the above patch, even though it doesn’t give you full PyPy support?

I think that would be ok, but you'll have to provide maintenance when
needed, because I think we're unlikely to make PyPy mandatory in our CI
setup.

Regards

Antoine.