You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by GitBox <gi...@apache.org> on 2022/08/03 10:48:25 UTC
[GitHub] [buildstream] ssssam opened a new issue, #1712: Distribute buildstream Wheel packages that contain BuildBox
ssssam opened a new issue, #1712:
URL: https://github.com/apache/buildstream/issues/1712
# Intro
Bst2 is available [in PyPI](https://pypi.org/project/BuildStream/), but only as a [source distribution](https://packaging.python.org/en/latest/specifications/source-distribution-format/). Many packages on PyPI can be installed simply with `pip3 install $package`. Installing BuildStream is not so simple:
* A working `cc` or `gcc` must be available, along with CPython headers and libraries, so Cython modules can be compiled to binaries
* BuildBox must be installed, this currently implies building from source ([see: buildbox issue about releasing binaries](https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/issues/1))
The Python [Wheel](https://peps.python.org/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl) format can be used to bundle and distribute binaries for our target platforms. When you run `pip3 install buildstream` and a suitable `buildstream-*.whl` is available on PyPI, the install should work even in a 'stock' container image:
* Cython modules are prebuilt so GCC isn't needed
* BuildBox binaries can be bundled in the wheel and installed in '$prefix/bin/' by Pip
# Goals
The goal is a way to install BuildStream 2 in a single command on a wide range of machines, assuming:
* the OS doesn't provide its own BuildStream package (we prefer distro packages where available)
* the user trusts PyPI and the BuildStream release team
Non goals:
* replace distro builds. Anyone redistributing BuildStream should use the sdist or Git repo and build from source.
# What about Docker?
The [Bst 1.6 install instructions](https://docs.buildstream.build/1.6/install_docker.html) recommend running in Docker via [bst-here](https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here). The [Bst 1.9.95 install instructions](https://docs.buildstream.build/1.95/main_install.html#install-container) also link to the [buildstream-docker-images instructions](https://gitlab.com/BuildStream/buildstream-docker-images/-/blob/master/USING.md). (At time of writing the [buildstream.build website](https://buildstream.build/install.html) doesn't mention Docker directly).
Provisioning Bst via a container image is sensible for many situations and we will always ensure it is possible. **Recommending it as the best way to use BuildStream** is more nuanced.
Advantages of recommending users obtain Bst via Docker:
* Single-command setup that "just works" for most cases
* We already build and publish usable Docker images at https://gitlab.com/BuildStream/buildstream-docker-images/-/issues
Disadvantages of recommending users obtain Bst via Docker:
* Dependency on hub.docker.io
* Requirement that user can start a container - not easy if they're already in a container, or don't have the necessary privileges
* Builds have different requirements for host tools depending on what plugins are used, so there's no 'one size fits all' container
* Most build tools don't require a separate container to run, so this makes `bst` look like a 'heavyweight' tool
* Docker images are larger than wheels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on issue #1712: Distribute buildstream Wheel packages that contain BuildBox
Posted by GitBox <gi...@apache.org>.
ssssam commented on issue #1712:
URL: https://github.com/apache/buildstream/issues/1712#issuecomment-1204117732
# Steps for "first cut"
* [x] Try cibuildwheel locally - done
* [x] Try cibuildwheel in GitHub Actions - done
* [ ] Automate releasing wheels to GitHub and PyPI
* [ ] Include BuildBox binaries in wheel
* Depends: https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/issues/1 (release BuildBox binaries)
* [ ] Ensure Bst uses the shipped buildbox binaries
# Additional tasks
* [ ] Investigate if we can and should strip debug info
* [ ] Investigate if we can build wheels for aarch64, riscv
* See https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation
* [ ] Auto updates of cibuildwheel version
* See https://cibuildwheel.readthedocs.io/en/stable/faq/#automatic-updates
* [ ] Revisit musl support - if buildbox binaries don't support musl, then don't ship wheels for musllinux platform either.
# Verification
* [ ] buildstream-2.0.0-....whl can be installed and used in a venv, with bst 1.x installed on the host
* [ ] buildstream-2.0.0.....whl can be installed on the host, with bst 1.x in a venv
* [ ] Wheel can be installed and used on a stock container image of Ubuntu, Fedora, Debian without needing extra packages
* [ ] Test each wheel that we release on the platform it expects to run (e.g. the python3.7 wheel on an OS with python3.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] ssssam commented on issue #1712: Distribute buildstream Wheel packages that contain BuildBox
Posted by GitBox <gi...@apache.org>.
ssssam commented on issue #1712:
URL: https://github.com/apache/buildstream/issues/1712#issuecomment-1205472939
Some progress on this in https://github.com/ssssam/buildstream/tree/sam/fat-wheels:
* using bundled BuildBox binaries works in my limited testing
* bundling the binaries into .whl packages works when calling setup.py directly, but not when using `pip` or `build` to produce a wheel. More investigation needed into why these tools all behave differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan closed issue #1712: Distribute buildstream Wheel packages that contain BuildBox
Posted by GitBox <gi...@apache.org>.
gtristan closed issue #1712: Distribute buildstream Wheel packages that contain BuildBox
URL: https://github.com/apache/buildstream/issues/1712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [buildstream] gtristan commented on issue #1712: Distribute buildstream Wheel packages that contain BuildBox
Posted by GitBox <gi...@apache.org>.
gtristan commented on issue #1712:
URL: https://github.com/apache/buildstream/issues/1712#issuecomment-1214740377
Note that this appears to be related to long standing epic #332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@buildstream.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org