You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Alessandro Molina <al...@ursacomputing.com> on 2021/08/16 15:08:10 UTC
[DISCUSS][Python] Moving Python specific code into pyarrow
PyArrow is currently full Cython codebase, but in reality it relies on some
classes and functions that are implemented in C++ within the src/python
directory ( https://github.com/apache/arrow/tree/master/cpp/src/arrow/python
). Especially for numpy/pandas conversion code that has to interface with
Numpy arrays data at low level.
When working in the area of PyArrow it's not uncommon that you end up
jumping back and forth between the Arrow C++ codebase for Python and
PyArrow and you can also end up with, sometimes hard to catch, integration
issues if you forgot to recompile libarrow even if you are working on a
Python only change.
I'm wondering if it wouldn't make life easier for contributors if the
src/arrow/python directory was moved into pyarrow and we made PyArrow able
to build it.
That would probably reduce risk of integration issues as rebuilding pyarrow
alone would probably be enough for most python specific changes (as it
would also rebuild the Python specific C++).
I think that moving src/arrow/python into pyarrow would also make the
codebase more cohesive which would lower the barrier for new contributors
looking for how to fix a pyarrow specific issue.
Unless there is any major side effect (outside of having to build a bit
more complex build scripts for pyarrow, but it's already CMake based, so
building some C++ shouldn't be a big deal) that I'm missing, it seems that
the benefits of having all Python related code into a single place would
surpass the side effects.
Also I'm not sure how widespread it is the requirement of Python from C++,
but it seems to me that if we moved all Python specific code into pyarrow
we could make libarrow decoupled from Python. Which might make it easier to
deal with Virtualenvs or debug versions of python as you wouldn't have to
deal with Python3_EXECUTABLE etc when building libarrow.
Any thoughts?
Re: [DISCUSS][Python] Moving Python specific code into pyarrow
Posted by Wes McKinney <we...@gmail.com>.
It seems like a good idea to attempt to make this change. The most
difficult thing might be projects that use the arrow/python/pyarrow.h
C++ API, so we would have to provide a viable migration path for
those. turbodbc is one example
https://github.com/blue-yonder/turbodbc/search?l=C%2B%2B&q=pyarrow.h
On Mon, Aug 16, 2021 at 6:00 PM Eduardo Ponce <ed...@gmail.com> wrote:
>
> I agree with this proposal, the Arrow C++ library does not need to depend
> on Python or PyArrow code.
> AFAIU this will eliminate the use of -DARROW_PYTHON build flag for Arrow
> C++ given that Python-related code will be compiled with PyArrow builds.
> Besides the use of "ARROW_PYTHON" env variable in CMakeLists.txt, the
> "dbi/hiveserver2" build makes use of "ARROW_PYTHON_SHARED_LINK_LIBS" [1].
>
> [1]
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/dbi/hiveserver2/CMakeLists.txt#L90
>
> ~Eduardo
>
> On Mon, Aug 16, 2021 at 11:24 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > I definitely think this is desirable.
> >
> > There's probably going to be a bit of work getting it to pass on all CI
> > (including the various nightly builds).
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 16/08/2021 à 17:08, Alessandro Molina a écrit :
> > > PyArrow is currently full Cython codebase, but in reality it relies on
> > some
> > > classes and functions that are implemented in C++ within the src/python
> > > directory (
> > https://github.com/apache/arrow/tree/master/cpp/src/arrow/python
> > > ). Especially for numpy/pandas conversion code that has to interface with
> > > Numpy arrays data at low level.
> > >
> > > When working in the area of PyArrow it's not uncommon that you end up
> > > jumping back and forth between the Arrow C++ codebase for Python and
> > > PyArrow and you can also end up with, sometimes hard to catch,
> > integration
> > > issues if you forgot to recompile libarrow even if you are working on a
> > > Python only change.
> > >
> > > I'm wondering if it wouldn't make life easier for contributors if the
> > > src/arrow/python directory was moved into pyarrow and we made PyArrow
> > able
> > > to build it.
> > >
> > > That would probably reduce risk of integration issues as rebuilding
> > pyarrow
> > > alone would probably be enough for most python specific changes (as it
> > > would also rebuild the Python specific C++).
> > >
> > > I think that moving src/arrow/python into pyarrow would also make the
> > > codebase more cohesive which would lower the barrier for new contributors
> > > looking for how to fix a pyarrow specific issue.
> > >
> > > Unless there is any major side effect (outside of having to build a bit
> > > more complex build scripts for pyarrow, but it's already CMake based, so
> > > building some C++ shouldn't be a big deal) that I'm missing, it seems
> > that
> > > the benefits of having all Python related code into a single place would
> > > surpass the side effects.
> > >
> > > Also I'm not sure how widespread it is the requirement of Python from
> > C++,
> > > but it seems to me that if we moved all Python specific code into pyarrow
> > > we could make libarrow decoupled from Python. Which might make it easier
> > to
> > > deal with Virtualenvs or debug versions of python as you wouldn't have to
> > > deal with Python3_EXECUTABLE etc when building libarrow.
> > >
> > > Any thoughts?
> > >
> >
Re: [DISCUSS][Python] Moving Python specific code into pyarrow
Posted by Eduardo Ponce <ed...@gmail.com>.
I agree with this proposal, the Arrow C++ library does not need to depend
on Python or PyArrow code.
AFAIU this will eliminate the use of -DARROW_PYTHON build flag for Arrow
C++ given that Python-related code will be compiled with PyArrow builds.
Besides the use of "ARROW_PYTHON" env variable in CMakeLists.txt, the
"dbi/hiveserver2" build makes use of "ARROW_PYTHON_SHARED_LINK_LIBS" [1].
[1]
https://github.com/apache/arrow/blob/master/cpp/src/arrow/dbi/hiveserver2/CMakeLists.txt#L90
~Eduardo
On Mon, Aug 16, 2021 at 11:24 AM Antoine Pitrou <an...@python.org> wrote:
>
> I definitely think this is desirable.
>
> There's probably going to be a bit of work getting it to pass on all CI
> (including the various nightly builds).
>
> Regards
>
> Antoine.
>
>
> Le 16/08/2021 à 17:08, Alessandro Molina a écrit :
> > PyArrow is currently full Cython codebase, but in reality it relies on
> some
> > classes and functions that are implemented in C++ within the src/python
> > directory (
> https://github.com/apache/arrow/tree/master/cpp/src/arrow/python
> > ). Especially for numpy/pandas conversion code that has to interface with
> > Numpy arrays data at low level.
> >
> > When working in the area of PyArrow it's not uncommon that you end up
> > jumping back and forth between the Arrow C++ codebase for Python and
> > PyArrow and you can also end up with, sometimes hard to catch,
> integration
> > issues if you forgot to recompile libarrow even if you are working on a
> > Python only change.
> >
> > I'm wondering if it wouldn't make life easier for contributors if the
> > src/arrow/python directory was moved into pyarrow and we made PyArrow
> able
> > to build it.
> >
> > That would probably reduce risk of integration issues as rebuilding
> pyarrow
> > alone would probably be enough for most python specific changes (as it
> > would also rebuild the Python specific C++).
> >
> > I think that moving src/arrow/python into pyarrow would also make the
> > codebase more cohesive which would lower the barrier for new contributors
> > looking for how to fix a pyarrow specific issue.
> >
> > Unless there is any major side effect (outside of having to build a bit
> > more complex build scripts for pyarrow, but it's already CMake based, so
> > building some C++ shouldn't be a big deal) that I'm missing, it seems
> that
> > the benefits of having all Python related code into a single place would
> > surpass the side effects.
> >
> > Also I'm not sure how widespread it is the requirement of Python from
> C++,
> > but it seems to me that if we moved all Python specific code into pyarrow
> > we could make libarrow decoupled from Python. Which might make it easier
> to
> > deal with Virtualenvs or debug versions of python as you wouldn't have to
> > deal with Python3_EXECUTABLE etc when building libarrow.
> >
> > Any thoughts?
> >
>
Re: [DISCUSS][Python] Moving Python specific code into pyarrow
Posted by Antoine Pitrou <an...@python.org>.
I definitely think this is desirable.
There's probably going to be a bit of work getting it to pass on all CI
(including the various nightly builds).
Regards
Antoine.
Le 16/08/2021 à 17:08, Alessandro Molina a écrit :
> PyArrow is currently full Cython codebase, but in reality it relies on some
> classes and functions that are implemented in C++ within the src/python
> directory ( https://github.com/apache/arrow/tree/master/cpp/src/arrow/python
> ). Especially for numpy/pandas conversion code that has to interface with
> Numpy arrays data at low level.
>
> When working in the area of PyArrow it's not uncommon that you end up
> jumping back and forth between the Arrow C++ codebase for Python and
> PyArrow and you can also end up with, sometimes hard to catch, integration
> issues if you forgot to recompile libarrow even if you are working on a
> Python only change.
>
> I'm wondering if it wouldn't make life easier for contributors if the
> src/arrow/python directory was moved into pyarrow and we made PyArrow able
> to build it.
>
> That would probably reduce risk of integration issues as rebuilding pyarrow
> alone would probably be enough for most python specific changes (as it
> would also rebuild the Python specific C++).
>
> I think that moving src/arrow/python into pyarrow would also make the
> codebase more cohesive which would lower the barrier for new contributors
> looking for how to fix a pyarrow specific issue.
>
> Unless there is any major side effect (outside of having to build a bit
> more complex build scripts for pyarrow, but it's already CMake based, so
> building some C++ shouldn't be a big deal) that I'm missing, it seems that
> the benefits of having all Python related code into a single place would
> surpass the side effects.
>
> Also I'm not sure how widespread it is the requirement of Python from C++,
> but it seems to me that if we moved all Python specific code into pyarrow
> we could make libarrow decoupled from Python. Which might make it easier to
> deal with Virtualenvs or debug versions of python as you wouldn't have to
> deal with Python3_EXECUTABLE etc when building libarrow.
>
> Any thoughts?
>
Re: [DISCUSS][Python] Moving Python specific code into pyarrow
Posted by Micah Kornfield <em...@gmail.com>.
This seems reasonable as long as it is actually feasible (the dependencies
are cleanly separable)..
A while ago I had a proof of concept bazel build working that was able to
automatically build the changes together.
On Monday, August 16, 2021, David Li <li...@apache.org> wrote:
> I support this. In the past I had to effectively do this manually to build
> Arrow/PyArrow in a monorepo (to build for multiple Python versions
> simultaneously without having conflicting copies of Arrow for each Python
> version). From what I remember, there's some usage of Arrow-internal
> headers that need to be replaced, but fortunately they were all very simple
> to replace.
>
> Though in my personal experience, it wasn't often that I needed to touch
> src/arrow/python.
>
> -David
>
> On Mon, Aug 16, 2021, at 11:08, Alessandro Molina wrote:
> > PyArrow is currently full Cython codebase, but in reality it relies on
> some
> > classes and functions that are implemented in C++ within the src/python
> > directory ( https://github.com/apache/arrow/tree/master/cpp/src/
> arrow/python
> > ). Especially for numpy/pandas conversion code that has to interface with
> > Numpy arrays data at low level.
> >
> > When working in the area of PyArrow it's not uncommon that you end up
> > jumping back and forth between the Arrow C++ codebase for Python and
> > PyArrow and you can also end up with, sometimes hard to catch,
> integration
> > issues if you forgot to recompile libarrow even if you are working on a
> > Python only change.
> >
> > I'm wondering if it wouldn't make life easier for contributors if the
> > src/arrow/python directory was moved into pyarrow and we made PyArrow
> able
> > to build it.
> >
> > That would probably reduce risk of integration issues as rebuilding
> pyarrow
> > alone would probably be enough for most python specific changes (as it
> > would also rebuild the Python specific C++).
> >
> > I think that moving src/arrow/python into pyarrow would also make the
> > codebase more cohesive which would lower the barrier for new contributors
> > looking for how to fix a pyarrow specific issue.
> >
> > Unless there is any major side effect (outside of having to build a bit
> > more complex build scripts for pyarrow, but it's already CMake based, so
> > building some C++ shouldn't be a big deal) that I'm missing, it seems
> that
> > the benefits of having all Python related code into a single place would
> > surpass the side effects.
> >
> > Also I'm not sure how widespread it is the requirement of Python from
> C++,
> > but it seems to me that if we moved all Python specific code into pyarrow
> > we could make libarrow decoupled from Python. Which might make it easier
> to
> > deal with Virtualenvs or debug versions of python as you wouldn't have to
> > deal with Python3_EXECUTABLE etc when building libarrow.
> >
> > Any thoughts?
> >
>
Re: [DISCUSS][Python] Moving Python specific code into pyarrow
Posted by David Li <li...@apache.org>.
I support this. In the past I had to effectively do this manually to build Arrow/PyArrow in a monorepo (to build for multiple Python versions simultaneously without having conflicting copies of Arrow for each Python version). From what I remember, there's some usage of Arrow-internal headers that need to be replaced, but fortunately they were all very simple to replace.
Though in my personal experience, it wasn't often that I needed to touch src/arrow/python.
-David
On Mon, Aug 16, 2021, at 11:08, Alessandro Molina wrote:
> PyArrow is currently full Cython codebase, but in reality it relies on some
> classes and functions that are implemented in C++ within the src/python
> directory ( https://github.com/apache/arrow/tree/master/cpp/src/arrow/python
> ). Especially for numpy/pandas conversion code that has to interface with
> Numpy arrays data at low level.
>
> When working in the area of PyArrow it's not uncommon that you end up
> jumping back and forth between the Arrow C++ codebase for Python and
> PyArrow and you can also end up with, sometimes hard to catch, integration
> issues if you forgot to recompile libarrow even if you are working on a
> Python only change.
>
> I'm wondering if it wouldn't make life easier for contributors if the
> src/arrow/python directory was moved into pyarrow and we made PyArrow able
> to build it.
>
> That would probably reduce risk of integration issues as rebuilding pyarrow
> alone would probably be enough for most python specific changes (as it
> would also rebuild the Python specific C++).
>
> I think that moving src/arrow/python into pyarrow would also make the
> codebase more cohesive which would lower the barrier for new contributors
> looking for how to fix a pyarrow specific issue.
>
> Unless there is any major side effect (outside of having to build a bit
> more complex build scripts for pyarrow, but it's already CMake based, so
> building some C++ shouldn't be a big deal) that I'm missing, it seems that
> the benefits of having all Python related code into a single place would
> surpass the side effects.
>
> Also I'm not sure how widespread it is the requirement of Python from C++,
> but it seems to me that if we moved all Python specific code into pyarrow
> we could make libarrow decoupled from Python. Which might make it easier to
> deal with Virtualenvs or debug versions of python as you wouldn't have to
> deal with Python3_EXECUTABLE etc when building libarrow.
>
> Any thoughts?
>