You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/04 21:35:42 UTC

[GitHub] [pulsar] zbentley opened a new pull request, #15436: [python] [build system] Improve MacOS Wheel Building with Pyenv and Rosetta

zbentley opened a new pull request, #15436:
URL: https://github.com/apache/pulsar/pull/15436

Fixes https://github.com/apache/pulsar/issues/15402

### Motivation

Building Python client wheels on MacOS is troublesome on some combinations of architecture/python version. Specifically, Python 3.7 is the most widely used version of Python 3, but it does not support building "universal" binary wheels (wheels which contain both ARM and Intel-architecture binary artifacts).

Additionally, building Python from source on MacOS is troublesome and prone to breakage in older (or future) MacOS versions.

As a result, the wheel building Bash automation refrains from creating 3.7 wheels on ARM, and otherwise special-cases 3.7 in a few different ways.

Two tools are available to help with these issues which the Pulsar client build system does not currently make use of:
- The [pyenv](https://github.com/pyenv/pyenv) project is a widely used means of getting around "python has trouble building from source" on many systems, including MacOS systems. They maintain a curated list of patches (mostly to Python's build system rather than Python itself) to allow it to build well on all sorts of environments. They also provide tooling for semi-hermetic "virtual python environments" (distinct from [virtualenvs](https://docs.python.org/3/library/venv.html) in that they encapsulate a separate Python _interpreter_, rather than just a set of dependencies). Using those environments, it's possible to build and install multiple versions of Python and switch between them.
- The [rosetta](https://developer.apple.com/documentation/apple-silicon/about-the-rosetta-translation-environment) translation system on ARM Macs allows those Macs to run binaries (including compiler toolchains) for both the ARM and x86 architecture.

### Modifications

This PR alters the wheel creation script in a few ways:
- The MacOS SDK (the stdlib and system headers/libraries) is now looked up using the standard `xcrun` command rather than via hardcoded and templated path.
- Python versions iterated over in the script are also qualified with their architecture--x86, arm, or universal.
- The OpenSSL version that is built for the Pulsar client to link against is also used by Python builds.
- `pyenv` is installed (via git master checkout) in the wheel building scratch environment. `git clone` is Pyenv's suggested installation mechanism, since many of their patches land in `master` before they are available in pyenv package artifacts via e.g. Homebrew or dpkg repositories.
- `pyenv` is used to install Pythons, rather than a direct download-and-compile. Appropriate flags are set/unset for each python/architecture "tuple". For example, for 3.7, only native-architecture, non-universal wheels can be created, so its build/install is run twice, one with an ARM architecture and once with x86. **Note that this means that arm64 python37 builds will fail on Intel Macs**; the Rosetta translation layer used to allow ARM macs to run the Intel build system does not reverse, and so the `arch` command's invocation will fail for that permutation on an Intel mac. I suggest that Intel mac compiling users comment that permutation out of the `PYTHON_VERSIONS` list.
- Within each pyenv, the `pip`, `setuptools`, and `wheel` packages are updated to their latest versions supported on the corresponding Python.
- Pyenv is also used to invoke `bdist_wheel` at the end, providing added assurance that wheel creation executes against the desired versions of resources.

### Verifying this change

I suggest the following test plan:

1. On a Mac running MacOS 11 or better, ensure `cmake` and the SDK are installed. Also install `pyenv` in a separate distribution for testing via `brew install pyenv`.
1. If your test system is an Intel Mac, comment out the python37/arm64 entry from the PYTHON_VERSIONS list in this PR's file.
1. Run `bash build-python-wheels.sh`
1. Ensure that the expected wheel artifacts corresponding to the listed Python versions are created in the `pulsar-client-cpp/python/dist` directory.
1. For each packaged version of Python, do `pyenv install $PYTHON_VERSION` to make that Python globally available on your Mac. You should *not* use the pyenv-installed pythons created by the wheel builder script.
1. For each python, do `pyenv global $PYTHON_VERSION`; ensure `python --version` reports the correct string.
1. For each python, do `python -mpip install --upgrade pip; python -mpip install --upgrade setuptools wheel`.
1. For each python, find the corresponding wheel file in `dist/` and do `python -mpip install /path/to/pulsar-client-wheel.whl`. Ensure it installs successfully.
1. Ensure that only system-avaialble or relative linker paths are available via `otool -L $(python -c 'import _pulsar; print(_pulsar.__file__)')`.
1. Run the python Pulsar client test suite against a running instance of Pulsar, and ensure all tests pass.

### Does this pull request potentially affect one of the following parts:

*If `yes` was chosen, please highlight the changes*

- Dependencies (does it add or upgrade a dependency): sort of
- The public API: no
- The schema: no
- The default values of configurations: no
- The wire protocol: no
- The rest endpoints: no
- The admin cli options: no
- Anything that affects deployment: client libraries

### Documentation

Need to update docs?

- [x] `no-need-doc`
The wheel creation process is unchanged externally.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org