You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Stestagg <st...@gmail.com> on 2019/07/04 20:10:10 UTC
linking 3rd party cython modules against pyarrow fails since 0.14.0
Hi
I've got a cython module that links against PyArrow, using the
'pyarrow.get_libraries()' associated methods.
Builds on Windows and Linux are consitently failing against 0.14, but
working on 0.12 to 0.13.
Linux gives:
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
build/temp.linux-x86_64-3.7/test.o
-L/usr/local/lib/python3.7/dist-packages/pyarrow -larrow -larrow_python -o
/home/dduser/att/arrowtest.cpython-37m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -larrow
/usr/bin/ld: cannot find -larrow_python
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-g++' failed with exit status 1
The windows build is more funky, but I'm still investigating.
A minimal example is:
setup.py:
import pyarrow
from Cython.Build import cythonize
from distutils.command.build_clib import build_clib
from distutils.core import setup, Extension
OPTIONS = {
'sources': ["test.pyx"],
'language': "c++",
'include_dirs': [pyarrow.get_include()],
'libraries': pyarrow.get_libraries(),
'library_dirs': pyarrow.get_library_dirs()
}
setup(
name='arrowtest',
ext_modules = cythonize(Extension("arrowtest",**OPTIONS)),
cmdclass = {'build_clib': build_clib},
version="0.1",
)
test.pyx:
import pyarrow as pa
cimport pyarrow.lib as pa
Thanks
Steve
Re: linking 3rd party cython modules against pyarrow fails since
0.14.0
Posted by Antoine Pitrou <an...@python.org>.
That's quite likely indeed.
A bit worrying is that this should have been caught by our unit tests.
Regards
Antoine.
Le 05/07/2019 à 10:02, Weston Steimel a écrit :
> Hello,
>
> I wonder if perhaps that may be due to the work done for reducing the wheel
> size in https://issues.apache.org/jira/browse/ARROW-5082?
>
> On Thu, Jul 4, 2019 at 10:06 PM Stestagg <st...@gmail.com> wrote:
>
>> 1) pip install pyarrow==0.14.0
>> 2) All the pyarrow files including, for example libarrow.so.14, but not
>> libarrow.so (hence the linker error)
>>
>> Reproducible on Python 3.7.2 on linux mint 19.1 and debian docker:
>>
>> Example dockerfile:
>> ```
>> FROM debian:unstable-slim
>>
>> RUN apt-get update && apt-get upgrade -y
>> RUN apt-get install -y python3 python3-dev
>> RUN apt-get install -y python3-pip
>>
>> RUN python3 -m pip install --upgrade pip
>> RUN pip3 install Cython pyarrow
>> COPY setup.py /root
>> COPY test.pyx /root
>> WORKDIR /root
>> RUN python3 setup.py build_ext --inplace
>> ```
>>
>> Where setup.py and test.pyx are the files listed above, with an added call
>> to numpy.get_include().
>>
>> Appending ' ==0.13.0' to the 'RUN pip3 install...' line above results in
>> the docker image building
>>
>> Steve
>>
>> On Thu, Jul 4, 2019 at 10:37 PM Antoine Pitrou <an...@python.org> wrote:
>>
>>>
>>> Hi,
>>>
>>> 1) How did you install PyArrow?
>>>
>>> 2) What does /usr/local/lib/python3.7/dist-packages/pyarrow contain?
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 04/07/2019 à 22:10, Stestagg a écrit :
>>>> Hi
>>>>
>>>> I've got a cython module that links against PyArrow, using the
>>>> 'pyarrow.get_libraries()' associated methods.
>>>>
>>>> Builds on Windows and Linux are consitently failing against 0.14, but
>>>> working on 0.12 to 0.13.
>>>>
>>>> Linux gives:
>>>> x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
>>>> -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat
>>>> -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
>>>> build/temp.linux-x86_64-3.7/test.o
>>>> -L/usr/local/lib/python3.7/dist-packages/pyarrow -larrow -larrow_python
>>> -o
>>>> /home/dduser/att/arrowtest.cpython-37m-x86_64-linux-gnu.so
>>>> /usr/bin/ld: cannot find -larrow
>>>> /usr/bin/ld: cannot find -larrow_python
>>>> collect2: error: ld returned 1 exit status
>>>> error: command 'x86_64-linux-gnu-g++' failed with exit status 1
>>>>
>>>> The windows build is more funky, but I'm still investigating.
>>>>
>>>> A minimal example is:
>>>>
>>>> setup.py:
>>>>
>>>> import pyarrow
>>>> from Cython.Build import cythonize
>>>> from distutils.command.build_clib import build_clib
>>>> from distutils.core import setup, Extension
>>>>
>>>>
>>>> OPTIONS = {
>>>> 'sources': ["test.pyx"],
>>>> 'language': "c++",
>>>> 'include_dirs': [pyarrow.get_include()],
>>>> 'libraries': pyarrow.get_libraries(),
>>>> 'library_dirs': pyarrow.get_library_dirs()
>>>> }
>>>>
>>>> setup(
>>>> name='arrowtest',
>>>> ext_modules = cythonize(Extension("arrowtest",**OPTIONS)),
>>>> cmdclass = {'build_clib': build_clib},
>>>> version="0.1",
>>>> )
>>>>
>>>> test.pyx:
>>>>
>>>> import pyarrow as pa
>>>> cimport pyarrow.lib as pa
>>>>
>>>> Thanks
>>>>
>>>> Steve
>>>>
>>>
>>
>
Re: linking 3rd party cython modules against pyarrow fails since 0.14.0
Posted by Weston Steimel <we...@gmail.com>.
Hello,
I wonder if perhaps that may be due to the work done for reducing the wheel
size in https://issues.apache.org/jira/browse/ARROW-5082?
On Thu, Jul 4, 2019 at 10:06 PM Stestagg <st...@gmail.com> wrote:
> 1) pip install pyarrow==0.14.0
> 2) All the pyarrow files including, for example libarrow.so.14, but not
> libarrow.so (hence the linker error)
>
> Reproducible on Python 3.7.2 on linux mint 19.1 and debian docker:
>
> Example dockerfile:
> ```
> FROM debian:unstable-slim
>
> RUN apt-get update && apt-get upgrade -y
> RUN apt-get install -y python3 python3-dev
> RUN apt-get install -y python3-pip
>
> RUN python3 -m pip install --upgrade pip
> RUN pip3 install Cython pyarrow
> COPY setup.py /root
> COPY test.pyx /root
> WORKDIR /root
> RUN python3 setup.py build_ext --inplace
> ```
>
> Where setup.py and test.pyx are the files listed above, with an added call
> to numpy.get_include().
>
> Appending ' ==0.13.0' to the 'RUN pip3 install...' line above results in
> the docker image building
>
> Steve
>
> On Thu, Jul 4, 2019 at 10:37 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hi,
> >
> > 1) How did you install PyArrow?
> >
> > 2) What does /usr/local/lib/python3.7/dist-packages/pyarrow contain?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 04/07/2019 à 22:10, Stestagg a écrit :
> > > Hi
> > >
> > > I've got a cython module that links against PyArrow, using the
> > > 'pyarrow.get_libraries()' associated methods.
> > >
> > > Builds on Windows and Linux are consitently failing against 0.14, but
> > > working on 0.12 to 0.13.
> > >
> > > Linux gives:
> > > x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> > > -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat
> > > -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
> > > build/temp.linux-x86_64-3.7/test.o
> > > -L/usr/local/lib/python3.7/dist-packages/pyarrow -larrow -larrow_python
> > -o
> > > /home/dduser/att/arrowtest.cpython-37m-x86_64-linux-gnu.so
> > > /usr/bin/ld: cannot find -larrow
> > > /usr/bin/ld: cannot find -larrow_python
> > > collect2: error: ld returned 1 exit status
> > > error: command 'x86_64-linux-gnu-g++' failed with exit status 1
> > >
> > > The windows build is more funky, but I'm still investigating.
> > >
> > > A minimal example is:
> > >
> > > setup.py:
> > >
> > > import pyarrow
> > > from Cython.Build import cythonize
> > > from distutils.command.build_clib import build_clib
> > > from distutils.core import setup, Extension
> > >
> > >
> > > OPTIONS = {
> > > 'sources': ["test.pyx"],
> > > 'language': "c++",
> > > 'include_dirs': [pyarrow.get_include()],
> > > 'libraries': pyarrow.get_libraries(),
> > > 'library_dirs': pyarrow.get_library_dirs()
> > > }
> > >
> > > setup(
> > > name='arrowtest',
> > > ext_modules = cythonize(Extension("arrowtest",**OPTIONS)),
> > > cmdclass = {'build_clib': build_clib},
> > > version="0.1",
> > > )
> > >
> > > test.pyx:
> > >
> > > import pyarrow as pa
> > > cimport pyarrow.lib as pa
> > >
> > > Thanks
> > >
> > > Steve
> > >
> >
>
Re: linking 3rd party cython modules against pyarrow fails since 0.14.0
Posted by Stestagg <st...@gmail.com>.
1) pip install pyarrow==0.14.0
2) All the pyarrow files including, for example libarrow.so.14, but not
libarrow.so (hence the linker error)
Reproducible on Python 3.7.2 on linux mint 19.1 and debian docker:
Example dockerfile:
```
FROM debian:unstable-slim
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y python3 python3-dev
RUN apt-get install -y python3-pip
RUN python3 -m pip install --upgrade pip
RUN pip3 install Cython pyarrow
COPY setup.py /root
COPY test.pyx /root
WORKDIR /root
RUN python3 setup.py build_ext --inplace
```
Where setup.py and test.pyx are the files listed above, with an added call
to numpy.get_include().
Appending ' ==0.13.0' to the 'RUN pip3 install...' line above results in
the docker image building
Steve
On Thu, Jul 4, 2019 at 10:37 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hi,
>
> 1) How did you install PyArrow?
>
> 2) What does /usr/local/lib/python3.7/dist-packages/pyarrow contain?
>
> Regards
>
> Antoine.
>
>
> Le 04/07/2019 à 22:10, Stestagg a écrit :
> > Hi
> >
> > I've got a cython module that links against PyArrow, using the
> > 'pyarrow.get_libraries()' associated methods.
> >
> > Builds on Windows and Linux are consitently failing against 0.14, but
> > working on 0.12 to 0.13.
> >
> > Linux gives:
> > x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> > -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat
> > -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
> > build/temp.linux-x86_64-3.7/test.o
> > -L/usr/local/lib/python3.7/dist-packages/pyarrow -larrow -larrow_python
> -o
> > /home/dduser/att/arrowtest.cpython-37m-x86_64-linux-gnu.so
> > /usr/bin/ld: cannot find -larrow
> > /usr/bin/ld: cannot find -larrow_python
> > collect2: error: ld returned 1 exit status
> > error: command 'x86_64-linux-gnu-g++' failed with exit status 1
> >
> > The windows build is more funky, but I'm still investigating.
> >
> > A minimal example is:
> >
> > setup.py:
> >
> > import pyarrow
> > from Cython.Build import cythonize
> > from distutils.command.build_clib import build_clib
> > from distutils.core import setup, Extension
> >
> >
> > OPTIONS = {
> > 'sources': ["test.pyx"],
> > 'language': "c++",
> > 'include_dirs': [pyarrow.get_include()],
> > 'libraries': pyarrow.get_libraries(),
> > 'library_dirs': pyarrow.get_library_dirs()
> > }
> >
> > setup(
> > name='arrowtest',
> > ext_modules = cythonize(Extension("arrowtest",**OPTIONS)),
> > cmdclass = {'build_clib': build_clib},
> > version="0.1",
> > )
> >
> > test.pyx:
> >
> > import pyarrow as pa
> > cimport pyarrow.lib as pa
> >
> > Thanks
> >
> > Steve
> >
>
Re: linking 3rd party cython modules against pyarrow fails since
0.14.0
Posted by Antoine Pitrou <an...@python.org>.
Hi,
1) How did you install PyArrow?
2) What does /usr/local/lib/python3.7/dist-packages/pyarrow contain?
Regards
Antoine.
Le 04/07/2019 à 22:10, Stestagg a écrit :
> Hi
>
> I've got a cython module that links against PyArrow, using the
> 'pyarrow.get_libraries()' associated methods.
>
> Builds on Windows and Linux are consitently failing against 0.14, but
> working on 0.12 to 0.13.
>
> Linux gives:
> x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat
> -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
> build/temp.linux-x86_64-3.7/test.o
> -L/usr/local/lib/python3.7/dist-packages/pyarrow -larrow -larrow_python -o
> /home/dduser/att/arrowtest.cpython-37m-x86_64-linux-gnu.so
> /usr/bin/ld: cannot find -larrow
> /usr/bin/ld: cannot find -larrow_python
> collect2: error: ld returned 1 exit status
> error: command 'x86_64-linux-gnu-g++' failed with exit status 1
>
> The windows build is more funky, but I'm still investigating.
>
> A minimal example is:
>
> setup.py:
>
> import pyarrow
> from Cython.Build import cythonize
> from distutils.command.build_clib import build_clib
> from distutils.core import setup, Extension
>
>
> OPTIONS = {
> 'sources': ["test.pyx"],
> 'language': "c++",
> 'include_dirs': [pyarrow.get_include()],
> 'libraries': pyarrow.get_libraries(),
> 'library_dirs': pyarrow.get_library_dirs()
> }
>
> setup(
> name='arrowtest',
> ext_modules = cythonize(Extension("arrowtest",**OPTIONS)),
> cmdclass = {'build_clib': build_clib},
> version="0.1",
> )
>
> test.pyx:
>
> import pyarrow as pa
> cimport pyarrow.lib as pa
>
> Thanks
>
> Steve
>