You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/07/08 13:36:00 UTC
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS
wheels depend on shared libs under /usr/local/opt
[ https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880355#comment-16880355 ]
Wes McKinney commented on ARROW-5874:
-------------------------------------
Thanks for the report. Since adding the OpenSSL dependency we expected some hiccups. There's a couple possible solutions
* Bundling OpenSSL shared libraries in the wheel
* Fixing the library path to not be looking for libraries in a hard-coded location (I thought that the wheel build was supposed to fix this)
* Only linking to OpenSSL in libarrow_flight (I'm not sure that it needs to be linked with {{pyarrow.lib}}). I think we should do this regardless
FYI: maintaining wheels is extremely difficult for us. The project, and the Python ecosystem in general, would be much better off if everyone used conda for complex packages like this
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
> -------------------------------------------------------------------------------
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
> Reporter: Michael Anselmi
> Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "<string>", line 1, in <module>
> # File "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py", line 49, in <module>
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError: dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so, 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from: /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been provided by Homebrew as part of your build process? Unfortunately this makes the pyarrow 0.14.0 wheel completely unusable on my system or any system that doesn't have OpenSSL installed in that location. This is a regression from pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}} and {{.so}} file in {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}} and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0, current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
> otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
> otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
> # @rpath/libarrow_python.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
> # /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1349.64.0)
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
> otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libgandiva.14.dylib
> # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libgandiva.14.dylib:
> # @rpath/libgandiva.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
> # /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
> otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libplasma.14.dylib
> # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libplasma.14.dylib:
> # @rpath/libplasma.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)