You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Vibhatha Abeykoon <vi...@gmail.com> on 2020/11/03 22:44:34 UTC
Development with C++ and Cython APIs in Arrow
Hello,
I have a question related to packaging an API written by using both C++ API
and Cython API of Arrow.
For now what I do is, build Arrow from source to generate both libarrow.so
and libarrow_python.so. When using the library, I have to point the
installed *.so using the LD_LIBRARY_PATH. But when packaging the project, I
am not quite sure whether this is the correct approach. For instance, when
generating a pip package, this workflow is not a good solution.
Any comments and suggestions?
With Regards,
Vibhatha Abeykoon,
Re: Development with C++ and Cython APIs in Arrow
Posted by Vibhatha Abeykoon <vi...@gmail.com>.
Thanks for these suggestions. Will take a look into these approaches.
With Regards,
Vibhatha Abeykoon
On Fri, Nov 6, 2020 at 2:43 PM Uwe L. Korn <uw...@xhochy.com> wrote:
> The pip package (explicitly the wheels) should contain the C++ libraries
> and headers. So it should be sufficient for your usecase and there
> shouldn't be a need for separately building the C++ artifacts.
>
> On Fri, Nov 6, 2020, at 5:18 PM, Vibhatha Abeykoon wrote:
> > One more question about packaging, here when the API requires both Cython
> > and C++ APIs,
> > Pyarrow dependency must also be built from the source? Or is it practical
> > to use the same version
> > of Arrow using Pip?
> >
> > With Regards,
> > Vibhatha Abeykoon
> >
> >
> > On Fri, Nov 6, 2020 at 9:59 AM Vibhatha Abeykoon <vi...@gmail.com>
> wrote:
> >
> > > Hello Uwe,
> > >
> > > Nice example. I will follow this.
> > >
> > > With Regards,
> > > Vibhatha Abeykoon
> > >
> > >
> > > On Fri, Nov 6, 2020 at 9:36 AM Uwe L. Korn <uw...@xhochy.com> wrote:
> > >
> > >> Hello Vibhatha,
> > >>
> > >> the best is to set a relative RPATH on the libraries. An example for
> this
> > >> can be seen in the turbodbc sources:
> > >>
> https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
> > >>
> > >> Cheers
> > >> Uwe
> > >>
> > >> On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
> > >> > Hello,
> > >> >
> > >> > I have a question related to packaging an API written by using both
> C++
> > >> API
> > >> > and Cython API of Arrow.
> > >> >
> > >> > For now what I do is, build Arrow from source to generate both
> > >> libarrow.so
> > >> > and libarrow_python.so. When using the library, I have to point the
> > >> > installed *.so using the LD_LIBRARY_PATH. But when packaging the
> > >> project, I
> > >> > am not quite sure whether this is the correct approach. For
> instance,
> > >> when
> > >> > generating a pip package, this workflow is not a good solution.
> > >> >
> > >> > Any comments and suggestions?
> > >> >
> > >> > With Regards,
> > >> > Vibhatha Abeykoon,
> > >> >
> > >>
> > >
> >
>
Re: Development with C++ and Cython APIs in Arrow
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
The pip package (explicitly the wheels) should contain the C++ libraries and headers. So it should be sufficient for your usecase and there shouldn't be a need for separately building the C++ artifacts.
On Fri, Nov 6, 2020, at 5:18 PM, Vibhatha Abeykoon wrote:
> One more question about packaging, here when the API requires both Cython
> and C++ APIs,
> Pyarrow dependency must also be built from the source? Or is it practical
> to use the same version
> of Arrow using Pip?
>
> With Regards,
> Vibhatha Abeykoon
>
>
> On Fri, Nov 6, 2020 at 9:59 AM Vibhatha Abeykoon <vi...@gmail.com> wrote:
>
> > Hello Uwe,
> >
> > Nice example. I will follow this.
> >
> > With Regards,
> > Vibhatha Abeykoon
> >
> >
> > On Fri, Nov 6, 2020 at 9:36 AM Uwe L. Korn <uw...@xhochy.com> wrote:
> >
> >> Hello Vibhatha,
> >>
> >> the best is to set a relative RPATH on the libraries. An example for this
> >> can be seen in the turbodbc sources:
> >> https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
> >>
> >> Cheers
> >> Uwe
> >>
> >> On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
> >> > Hello,
> >> >
> >> > I have a question related to packaging an API written by using both C++
> >> API
> >> > and Cython API of Arrow.
> >> >
> >> > For now what I do is, build Arrow from source to generate both
> >> libarrow.so
> >> > and libarrow_python.so. When using the library, I have to point the
> >> > installed *.so using the LD_LIBRARY_PATH. But when packaging the
> >> project, I
> >> > am not quite sure whether this is the correct approach. For instance,
> >> when
> >> > generating a pip package, this workflow is not a good solution.
> >> >
> >> > Any comments and suggestions?
> >> >
> >> > With Regards,
> >> > Vibhatha Abeykoon,
> >> >
> >>
> >
>
Re: Development with C++ and Cython APIs in Arrow
Posted by Maarten Breddels <ma...@gmail.com>.
Another approach I took is;
https://github.com/vaexio/vaex-arrow-ext
But it uses pybind11, not cython.
(from mobile phone)
On Fri, 6 Nov 2020, 17:19 Vibhatha Abeykoon, <vi...@gmail.com> wrote:
> One more question about packaging, here when the API requires both Cython
> and C++ APIs,
> Pyarrow dependency must also be built from the source? Or is it practical
> to use the same version
> of Arrow using Pip?
>
> With Regards,
> Vibhatha Abeykoon
>
>
> On Fri, Nov 6, 2020 at 9:59 AM Vibhatha Abeykoon <vi...@gmail.com>
> wrote:
>
> > Hello Uwe,
> >
> > Nice example. I will follow this.
> >
> > With Regards,
> > Vibhatha Abeykoon
> >
> >
> > On Fri, Nov 6, 2020 at 9:36 AM Uwe L. Korn <uw...@xhochy.com> wrote:
> >
> >> Hello Vibhatha,
> >>
> >> the best is to set a relative RPATH on the libraries. An example for
> this
> >> can be seen in the turbodbc sources:
> >>
> https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
> >>
> >> Cheers
> >> Uwe
> >>
> >> On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
> >> > Hello,
> >> >
> >> > I have a question related to packaging an API written by using both
> C++
> >> API
> >> > and Cython API of Arrow.
> >> >
> >> > For now what I do is, build Arrow from source to generate both
> >> libarrow.so
> >> > and libarrow_python.so. When using the library, I have to point the
> >> > installed *.so using the LD_LIBRARY_PATH. But when packaging the
> >> project, I
> >> > am not quite sure whether this is the correct approach. For instance,
> >> when
> >> > generating a pip package, this workflow is not a good solution.
> >> >
> >> > Any comments and suggestions?
> >> >
> >> > With Regards,
> >> > Vibhatha Abeykoon,
> >> >
> >>
> >
>
Re: Development with C++ and Cython APIs in Arrow
Posted by Vibhatha Abeykoon <vi...@gmail.com>.
One more question about packaging, here when the API requires both Cython
and C++ APIs,
Pyarrow dependency must also be built from the source? Or is it practical
to use the same version
of Arrow using Pip?
With Regards,
Vibhatha Abeykoon
On Fri, Nov 6, 2020 at 9:59 AM Vibhatha Abeykoon <vi...@gmail.com> wrote:
> Hello Uwe,
>
> Nice example. I will follow this.
>
> With Regards,
> Vibhatha Abeykoon
>
>
> On Fri, Nov 6, 2020 at 9:36 AM Uwe L. Korn <uw...@xhochy.com> wrote:
>
>> Hello Vibhatha,
>>
>> the best is to set a relative RPATH on the libraries. An example for this
>> can be seen in the turbodbc sources:
>> https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
>>
>> Cheers
>> Uwe
>>
>> On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
>> > Hello,
>> >
>> > I have a question related to packaging an API written by using both C++
>> API
>> > and Cython API of Arrow.
>> >
>> > For now what I do is, build Arrow from source to generate both
>> libarrow.so
>> > and libarrow_python.so. When using the library, I have to point the
>> > installed *.so using the LD_LIBRARY_PATH. But when packaging the
>> project, I
>> > am not quite sure whether this is the correct approach. For instance,
>> when
>> > generating a pip package, this workflow is not a good solution.
>> >
>> > Any comments and suggestions?
>> >
>> > With Regards,
>> > Vibhatha Abeykoon,
>> >
>>
>
Re: Development with C++ and Cython APIs in Arrow
Posted by Vibhatha Abeykoon <vi...@gmail.com>.
Hello Uwe,
Nice example. I will follow this.
With Regards,
Vibhatha Abeykoon
On Fri, Nov 6, 2020 at 9:36 AM Uwe L. Korn <uw...@xhochy.com> wrote:
> Hello Vibhatha,
>
> the best is to set a relative RPATH on the libraries. An example for this
> can be seen in the turbodbc sources:
> https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
>
> Cheers
> Uwe
>
> On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
> > Hello,
> >
> > I have a question related to packaging an API written by using both C++
> API
> > and Cython API of Arrow.
> >
> > For now what I do is, build Arrow from source to generate both
> libarrow.so
> > and libarrow_python.so. When using the library, I have to point the
> > installed *.so using the LD_LIBRARY_PATH. But when packaging the
> project, I
> > am not quite sure whether this is the correct approach. For instance,
> when
> > generating a pip package, this workflow is not a good solution.
> >
> > Any comments and suggestions?
> >
> > With Regards,
> > Vibhatha Abeykoon,
> >
>
Re: Development with C++ and Cython APIs in Arrow
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello Vibhatha,
the best is to set a relative RPATH on the libraries. An example for this can be seen in the turbodbc sources: https://github.com/blue-yonder/turbodbc/blob/80a29a7edfbdabf12410af01c0c0ae74bfc3aab4/setup.py#L186-L189
Cheers
Uwe
On Tue, Nov 3, 2020, at 11:44 PM, Vibhatha Abeykoon wrote:
> Hello,
>
> I have a question related to packaging an API written by using both C++ API
> and Cython API of Arrow.
>
> For now what I do is, build Arrow from source to generate both libarrow.so
> and libarrow_python.so. When using the library, I have to point the
> installed *.so using the LD_LIBRARY_PATH. But when packaging the project, I
> am not quite sure whether this is the correct approach. For instance, when
> generating a pip package, this workflow is not a good solution.
>
> Any comments and suggestions?
>
> With Regards,
> Vibhatha Abeykoon,
>