You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/05 08:18:52 UTC
[GitHub] [pulsar] BewareMyPower opened a new issue, #17468: [Bug] Cannot build Python wheels for macOS
BewareMyPower opened a new issue, #17468:
URL: https://github.com/apache/pulsar/issues/17468
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
- OS: macOS Monterey 12.3.1
- CMake version: 3.21.1
- Clang++ version: Apple clang version 13.1.6 (clang-1316.0.21.2.5)
- Python version: 3.10.6
- Pulsar version: v2.10.1
### Minimal reproduce step
Run the following command.
```bash
pulsar-client-cpp/python/build-mac-wheels.sh
```
### What did you expect to see?
The Python wheels for macOS should be generated under `pulsar-client-cpp/python/pkg/osx/` directoyr.
### What did you see instead?
The error logs:
```
DYLD_LIBRARY_PATH=/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13 ./python.exe -E ./setup.py install \
--prefix=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7 \
--install-scripts=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/bin \
--install-platlib=/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/lib/python3.7/lib-dynload \
--root=/
running install
running build
running build_ext
building '_lzma' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -fPIC -O3 -mmacosx-version-min=10.15 -I/Users/xuyunze/.pulsar-mac-wheels-cache/install/include -arch x86_64 -I/usr/local/opt/krb5/include -I./Include -I/Users/xuyunze/.pulsar-mac-wheels-cache/py-3.7/include -I. -I/usr/local/opt/krb5/include -I/usr/local/include -I/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Include -I/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13 -c /Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.c -o build/temp.macosx-10.15-x86_64-3.7/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.o
In file included from /Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.7.13/Modules/_lzmamodule.c:17:
/usr/local/include/lzma.h:293:10: fatal error: 'lzma/version.h' file not found
#include "lzma/version.h"
```
### Anything else?
This is a reason that the Python client 2.10.1 for macOS failed to release. @codelipenghui @tisonkun
I didn't look deeper into the script at the moment, but it looks like this script has some requirements for the OS version and Python version because it just builds the specific Python from source code in local env (not docker env). Maybe we need to enhance this script for more systems. /cc @merlimat
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1314658944
@tisonkun Yes. Close this issue by https://github.com/apache/pulsar/issues/17468#issuecomment-1284912921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1284989651
> @Jason918 It's already fixed in latest pulsar-client-python repo. The root cause is when building universal2 Python from source, it relies on the universal binary of zlib. So a universal binary of zlib must be built before building Python.
@BewareMyPower Great, I will try it on 2.10.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253697442
```
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```
- [out.log](https://github.com/apache/pulsar/files/9616855/out.log)
- [err.log](https://github.com/apache/pulsar/files/9616856/err.log)
The error logs are still the same as before. I think it's because some C dependencies must be installed when building Python from source. The script might require users to install some dependencies via `brew install`.
For example, the `'lzma/version.h' file not found` error requires the lzma dependency, so I searched for the related headers in `/usr` directory.
```bash
$ find /usr -name "*.h" | grep lzma
find: /usr/sbin/authserver: Permission denied
/usr/local/include/lzma.h
/usr/local/Cellar/xz/5.2.6/include/lzma.h
/usr/local/Cellar/xz/5.2.6/include/lzma/index.h
/usr/local/Cellar/xz/5.2.6/include/lzma/version.h
/usr/local/Cellar/xz/5.2.6/include/lzma/index_hash.h
/usr/local/Cellar/xz/5.2.6/include/lzma/lzma12.h
/usr/local/Cellar/xz/5.2.6/include/lzma/container.h
/usr/local/Cellar/xz/5.2.6/include/lzma/delta.h
/usr/local/Cellar/xz/5.2.6/include/lzma/vli.h
/usr/local/Cellar/xz/5.2.6/include/lzma/check.h
/usr/local/Cellar/xz/5.2.6/include/lzma/bcj.h
/usr/local/Cellar/xz/5.2.6/include/lzma/stream_flags.h
/usr/local/Cellar/xz/5.2.6/include/lzma/block.h
/usr/local/Cellar/xz/5.2.6/include/lzma/hardware.h
/usr/local/Cellar/xz/5.2.6/include/lzma/filter.h
/usr/local/Cellar/xz/5.2.6/include/lzma/base.h
```
`lzma` is included in `xz` dependency in macOS. But the install path is `/usr/local/Cellar/xz/<version>`, the include directory and library directory are not added to the related environment variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1284905484
> The root cause is the `--with-universal-archs` option. When it's `universal2` on a macOS with Intel based CPU architecture, `make test` would fail. But if it's `Intel-64`, the Python wheel might not support ARM architecture. Currently, I will change the option to `Intel-64` to make the script work for Intel based macs.
Same here when I am releasing 2.10.2.
Seems the best option now would be not supporting ARM until it's fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1290234277
> So a universal binary of zlib must be built before building Python.
Got error when compiling zlib. See the log:
[build-mac.log](https://github.com/apache/pulsar/files/9858792/build-mac.log)
Version
OS: MacOS 10.15.7
CMake version: 3.24.2
Clang version: Apple clang version 11.0.3 (clang-1103.0.32.59)
Pulsar version: v2.10.2
@BewareMyPower Can you help release python mac wheels for release 2.10.2?
From the "Existing branches" part in https://github.com/apache/pulsar/issues/17724, we need to keep release pulsar-client-python from main repo for active branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253307771
Confirm that Python support Apple M1 silicon arch from 3.9: https://docs.python.org/3/whatsnew/3.9.html#macos-11-0-big-sur-and-apple-silicon-mac-support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253321351
@tisonkun It still doesn't work.
```
dyld[46563]: missing symbol called
Traceback (most recent call last):
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__main__.py", line 5, in <module>
sys.exit(ensurepip._main())
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 277, in _main
return _bootstrap(
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 193, in _bootstrap
return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/ensurepip/__init__.py", line 93, in _run_pip
return subprocess.run([sys.executable, '-W', 'ignore::DeprecationWarning',
File "/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/Lib/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/xuyunze/.pulsar-mac-wheels-cache/Python-3.10.2/python.exe', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933/setuptools-58.1.0-py3-none-any.whl\', \'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933/pip-21.2.4-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/var/folders/b5/5vrry5193db1v8r7xfg6r3880000gn/T/tmp9_b1n933\', \'--root\', \'/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.
make: *** [install] Error 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253313669
Thank you @tisonkun, I'm testing it with 3.10 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253298596
@BewareMyPower I'm running the `build-mac-wheels.sh` script on macOS with Apple M1 silicon. It seems that old versions of Python don't support new arch.
After comment out the old version:
```diff
diff --git a/pulsar-client-cpp/python/build-mac-wheels.sh b/pulsar-client-cpp/python/build-mac-wheels.sh
index 6a4dae7117..21e37ca887 100755
--- a/pulsar-client-cpp/python/build-mac-wheels.sh
+++ b/pulsar-client-cpp/python/build-mac-wheels.sh
@@ -21,9 +21,9 @@
set -e
PYTHON_VERSIONS=(
- '3.7 3.7.13'
- '3.8 3.8.13'
- '3.9 3.9.10'
+# '3.7 3.7.13'
+# '3.8 3.8.13'
+# '3.9 3.9.10'
'3.10 3.10.2'
)
```
I build `pulsar_client-2.11.0-cp310-cp310-macosx_10_15_universal2.whl` successfully to `pulsar-client-cpp/python/dist`.
You may try to reproduce and I suggest we release more Python client as possible for resolvin #15477 & #16849. Perhaps no supports for old Python versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1313914371
@BewareMyPower is it fixed now in the separate repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower closed issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower closed issue #17468: [Bug] Cannot build Python wheels for macOS
URL: https://github.com/apache/pulsar/issues/17468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
tisonkun commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1253505558
@BewareMyPower can you attach the full log file and tell me what branch you're running against?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1284912921
@Jason918 It's already fixed in latest pulsar-client-python repo. The root cause is when building universal2 Python from source, it relies on the universal binary of zlib. So a universal binary of zlib must be built before building Python.
https://github.com/apache/pulsar-client-python/blob/main/pkg/mac/build-dependencies.sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #17468: [Bug] Cannot build Python wheels for macOS
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #17468:
URL: https://github.com/apache/pulsar/issues/17468#issuecomment-1290248228
@Jason918 I might not have time recently. The 2.8.4, 2.10.1 released also missed the Python mac wheels. IMO, for macOS users, they should turn to the new repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org