You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2022/08/17 08:13:44 UTC
[buildstream] branch tristan/build-wheels updated (9636aa813 -> d078c88c0)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
discard 9636aa813 tox: Remove obsolete '-e release' mode
discard 29d1d09c6 docs: Update release process
discard 8dd3c2d81 Mark unstable BuildStream releases with PEP440-compatible postfix
discard 163c65370 Build and release wheels packages as part of GitHub Actions
discard bb904ec6b .github/workflows/merge.yml: Rename 'build' and 'publish'
discard 39b391693 Add option to bundle BuildBox binaries inside Python package
new 1b612316d Add option to bundle BuildBox binaries inside Python package
new 0d31531ce .github/workflows/merge.yml: Rename 'build' and 'publish'
new a24be10ee Build and release wheels packages as part of GitHub Actions
new 37e3b314f Mark unstable BuildStream releases with PEP440-compatible postfix
new b90c4c64c docs: Update release process
new d078c88c0 tox: Remove obsolete '-e release' mode
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9636aa813)
\
N -- N -- N refs/heads/tristan/build-wheels (d078c88c0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/wheel-helpers/fetch-latest-buildbox-release.sh | 2 +-
setup.py | 5 ++++-
2 files changed, 5 insertions(+), 2 deletions(-)
[buildstream] 05/06: docs: Update release process
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit b90c4c64c83223d8fa70f42232050a9286f5dfdb
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Mon Aug 15 11:27:14 2022 +0200
docs: Update release process
---
doc/source/hacking/making_releases.rst | 18 ++++++++----------
1 file changed, 8 insertions(+), 10 deletions(-)
diff --git a/doc/source/hacking/making_releases.rst b/doc/source/hacking/making_releases.rst
index d9d34ae32..52d4890f0 100644
--- a/doc/source/hacking/making_releases.rst
+++ b/doc/source/hacking/making_releases.rst
@@ -173,6 +173,11 @@ Release process
git tag -s 1.1.1
git push origin 1.1.1
+ This will trigger the "Release actions" workflow which also takes care of:
+
+ * uploading Github release artifacts
+ * uploading Python source and binary packages to PyPI
+
* Upload the release tarball
First get yourself into a clean repository state, ensure that you
@@ -213,16 +218,9 @@ Post-release activities
Once the release has been published, there are some activities
which need to be done to ensure everything is up to date.
-* If this is a stable release, then the tarball should also be
- uploaded to PyPI.
-
- Make sure you have ``twine`` installed and upload the tarball
- like so:
-
- .. code:: shell
-
- pip3 install --user twine
- twine upload -r pypi dist/BuildStream-1.0.1.tar.gz
+* Check that the release was successfully uploaded to PyPI. If
+ it is an unstable release, make sure the version on PyPI has
+ the correct "dev0" postfix so to avoid being treated as stable.
* Update the topic line in the #buildstream IRC channel if needed
[buildstream] 06/06: tox: Remove obsolete '-e release' mode
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d078c88c0a0f25e794f83a243aa6aa7a0642283b
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Mon Aug 15 11:27:23 2022 +0200
tox: Remove obsolete '-e release' mode
---
tox.ini | 21 ---------------------
1 file changed, 21 deletions(-)
diff --git a/tox.ini b/tox.ini
index 33462b785..8d21de4f1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -208,27 +208,6 @@ deps =
-rrequirements/requirements.txt
-#
-# Publish a release on PyPI.
-#
-# This can be run manually but is useful to have here just to document how it's done,
-# it's also easier to just run:
-#
-# TWINE_USERNAME=buildstream TWINE_PASSWORD=<password> tox -e release
-#
-[testenv:release]
-skip_install = true
-commands =
- python3 setup.py sdist bdist_wheel
- twine upload -r pypi dist/*
-deps =
- twine
- wheel
-passenv =
- TWINE_USERNAME
- TWINE_PASSWORD
-
-
#
# Usefull for running arbitrary scripts in a BuildStream virtual env
#
[buildstream] 04/06: Mark unstable BuildStream releases with PEP440-compatible postfix
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 37e3b314f6592e63b9ca81365d3bae9c7ee978a0
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Fri Aug 12 12:22:55 2022 +0200
Mark unstable BuildStream releases with PEP440-compatible postfix
PyPI and Pip use the PEP440 version standard, which uses explicit
'.dev', '.a', '.b' and '.rc' suffixes to mark "unstable" releases.
BuildStream uses the even/odd convention. When running `pip install`
we want Pip to prefer a stable 2.0.0 release over an unstable 2.1.0
release. To make this work, all unstable releases now have a .dev0
suffix automatically added in setup.py, e.g. `2.1.0.dev0`.
This is necessary for us to fully automate releases to PyPI from GitHub
Actions.
---
pyproject.toml | 3 ++-
setup.cfg | 2 +-
setup.py | 42 +++++++++++++++++++++++++++++++++++++++++-
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/pyproject.toml b/pyproject.toml
index f3eadb7af..ea9f0df05 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,8 @@ requires = [
"setuptools>=36.6.0",
# In order to build wheels, and as required by PEP 517
"wheel",
- "Cython"
+ "Cython",
+ "packaging",
]
build-backend = "setuptools.build_meta"
diff --git a/setup.cfg b/setup.cfg
index aea4bdcc8..d220e3952 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
[versioneer]
VCS = git
-style = pep440
+style = pep440_buildstream
versionfile_source = src/buildstream/_version.py
versionfile_build = buildstream/_version.py
tag_prefix =
diff --git a/setup.py b/setup.py
index aeab34392..8e1a11556 100755
--- a/setup.py
+++ b/setup.py
@@ -23,6 +23,8 @@ from pathlib import Path
import re
import sys
+import packaging.version
+
###################################
# Ensure we have a version number #
@@ -32,11 +34,49 @@ import sys
sys.path.append(os.path.dirname(__file__))
import versioneer # pylint: disable=wrong-import-position
+
+def mark_unstable_version(version_string):
+ # When publishing to PyPI we must be sure that unstable releases are
+ # marked as such, so `pip install` doesn't install them by default.
+
+ v = packaging.version.parse(version_string)
+
+ # BuildStream version scheme: if MINOR version is odd, then
+ # this is an unstable release.
+ is_unstable_release = v.minor % 2 != 0
+
+ # Python PEP440 version scheme: use an explicit postfix to mark development
+ # and prereleases.
+ if is_unstable_release:
+ if v.local or v.is_devrelease or v.is_prerelease:
+ # PyPI will ignore these without us marking them.
+ return version_string
+ else:
+ return version_string + ".dev0"
+
+ return version_string
+
+
+# Extend versioneer to support our custom version style.
+_render = versioneer.render
+
+
+def render_version(pieces, style):
+ if style == "pep440_buildstream":
+ result = _render(pieces, "pep440")
+ result["version"] = mark_unstable_version(result["version"])
+ else:
+ result = _render(pieces, style)
+ return result
+
+
+versioneer.render = render_version
+
version = versioneer.get_version()
if version.startswith("0+untagged"):
print(
- "Your git repository has no tags - BuildStream can't " "determine its version. Please run `git fetch --tags`.",
+ "Your git repository has no tags - BuildStream can't determine its version. Please run `git fetch --tags`.",
file=sys.stderr,
)
sys.exit(1)
[buildstream] 01/06: Add option to bundle BuildBox binaries inside Python package
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 1b612316d9e9d6596ca5794538600e68cb381fb3
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Wed Aug 3 14:01:44 2022 +0200
Add option to bundle BuildBox binaries inside Python package
BuildBox is not widely distributed in binary form yet. For convience,
add a mechanism to bundle prebuilt binaries in the Python wheel
packages.
Setting BST_BUNDLE_BUILDBOX=1 when setup.py runs, causes the bundled
binaries to be included in the binary package. Its up to the packager
to make appropriate binaries available in
`src/buildstream/subprojects/buildbox/`.
BuildStream will search the package subprojects/ dir when looking for
BuildBox binaries on the host in all cases, prioritizing any bundled
binaries above other ones on the host.
Related to #1712
---
MANIFEST.in | 3 ++
pyproject.toml | 23 ++++++++++
setup.py | 60 +++++++++++++++++++++++++-
src/buildstream/_cas/casdprocessmanager.py | 7 ++-
src/buildstream/_site.py | 3 ++
src/buildstream/_testing/_utils/site.py | 4 +-
src/buildstream/sandbox/_sandboxbuildboxrun.py | 8 +++-
src/buildstream/utils.py | 42 ++++++++++++++----
tests/integration/cachedfail.py | 2 +-
tests/sandboxes/selection.py | 2 +-
10 files changed, 137 insertions(+), 17 deletions(-)
diff --git a/MANIFEST.in b/MANIFEST.in
index 71d24737c..9c28ca355 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -47,3 +47,6 @@ include versioneer.py
# setuptools.build_meta don't include setup.py by default. Add it
include setup.py
+
+# bundled binaries should only be in the bdist packages
+recursive-exclude src/buildstream/subprojects *
diff --git a/pyproject.toml b/pyproject.toml
index fefbbecd4..f3eadb7af 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,3 +24,26 @@ exclude = '''
| src/buildstream/_protos
)
'''
+
+[tool.cibuildwheel]
+build-frontend = "build"
+environment = { BST_BUNDLE_BUILDBOX = "1" }
+
+# The BuildBox binaries produced in buildbox-integration are linked against GLIBC 2.28
+# from Debian 10. See: https://gitlab.com/BuildGrid/buildbox/buildbox-integration.
+#
+# The PyPA manylinux_2_28 platform tag identifies that our wheel will run on any x86_64
+# OS with GLIBC >= 2.28. Following this setting, `cibuildwheel` builds the packages in
+# the corresponding manylinux_2_28 container image. See: https://github.com/pypa/manylinux
+manylinux-x86_64-image = "manylinux_2_28"
+
+skip = [
+ # BuildStream supports Python >= 3.7
+ "cp36-*",
+ # PyPy may work, but nobody is testing it so avoid distributing prebuilt binaries.
+ "pp*",
+ # Skipping this niche archicture ~halves overall build time.
+ "*_i686",
+ # The prebuilt BuildBox binaries link against GLibc so will work on manylinux but not musllinux
+ "*-musllinux_*",
+]
diff --git a/setup.py b/setup.py
index ade004938..aeab34392 100755
--- a/setup.py
+++ b/setup.py
@@ -64,6 +64,63 @@ except ImportError:
sys.exit(1)
+############################################################
+# List the BuildBox binaries to ship in the wheel packages #
+############################################################
+#
+# BuildBox isn't widely available in OS distributions. To enable a "one click"
+# install for BuildStream, we bundle prebuilt BuildBox binaries in our binary
+# wheel packages.
+#
+# The binaries are provided by the buildbox-integration Gitlab project:
+# https://gitlab.com/BuildGrid/buildbox/buildbox-integration
+#
+# If you want to build a wheel with the BuildBox binaries included, set the
+# env var "BST_BUNDLE_BUILDBOX=1" when running setup.py.
+
+try:
+ BUNDLE_BUILDBOX = int(os.environ.get("BST_BUNDLE_BUILDBOX", "0"))
+except ValueError:
+ print("BST_BUNDLE_BUILDBOX must be an integer. Please set it to '1' to enable, '0' to disable", file=sys.stderr)
+ raise SystemExit(1)
+
+
+def list_buildbox_binaries():
+ expected_binaries = [
+ "buildbox-casd",
+ "buildbox-fuse",
+ "buildbox-run",
+ ]
+
+ if BUNDLE_BUILDBOX:
+ bst_package_dir = Path(__file__).parent.joinpath("src/buildstream")
+ buildbox_dir = bst_package_dir.joinpath("subprojects", "buildbox")
+ buildbox_binaries = [buildbox_dir.joinpath(name) for name in expected_binaries]
+
+ missing_binaries = [path for path in buildbox_binaries if not path.is_file()]
+ if missing_binaries:
+ paths_text = "\n".join([" * {}".format(path) for path in missing_binaries])
+ print(
+ "Expected BuildBox binaries were not found. "
+ "Set BST_BUNDLE_BUILDBOX=0 or provide:\n\n"
+ "{}\n".format(paths_text),
+ file=sys.stderr,
+ )
+ raise SystemExit(1)
+
+ for path in buildbox_binaries:
+ if path.is_symlink():
+ print(
+ "Bundled BuildBox binaries must not be symlinks. Please fix {}".format(path),
+ file=sys.stderr,
+ )
+ raise SystemExit(1)
+
+ return [str(path.relative_to(bst_package_dir)) for path in buildbox_binaries]
+ else:
+ return []
+
+
###########################################
# List the pre-built man pages to install #
###########################################
@@ -351,7 +408,7 @@ setup(
},
python_requires="~={}.{}".format(REQUIRED_PYTHON_MAJOR, REQUIRED_PYTHON_MINOR),
package_dir={"": "src"},
- packages=find_packages(where="src", exclude=("tests", "tests.*")),
+ packages=find_packages(where="src", exclude=("subprojects", "tests", "tests.*")),
package_data={
"buildstream": [
"py.typed",
@@ -359,6 +416,7 @@ setup(
"plugins/*/*.yaml",
"data/*.yaml",
"data/*.sh.in",
+ *list_buildbox_binaries(),
*list_testing_datafiles(),
]
},
diff --git a/src/buildstream/_cas/casdprocessmanager.py b/src/buildstream/_cas/casdprocessmanager.py
index e7a73c1b2..2b702b925 100644
--- a/src/buildstream/_cas/casdprocessmanager.py
+++ b/src/buildstream/_cas/casdprocessmanager.py
@@ -73,7 +73,7 @@ class CASDProcessManager:
# Early version check
self._check_casd_version(messenger)
- casd_args = [utils.get_host_tool("buildbox-casd")]
+ casd_args = [self.__buildbox_casd()]
casd_args.append("--bind=" + self._connection_string)
casd_args.append("--log-level=" + log_level.value)
@@ -106,6 +106,9 @@ class CASDProcessManager:
casd_args, cwd=path, stdout=logfile_fp, stderr=subprocess.STDOUT, preexec_fn=os.setpgrp
)
+ def __buildbox_casd(self):
+ return utils._get_host_tool_internal("buildbox-casd", search_subprojects_dir="buildbox")
+
# _check_casd_version()
#
# Check for minimal acceptable version of buildbox-casd.
@@ -121,7 +124,7 @@ class CASDProcessManager:
# We specify a trailing "path" argument because some versions of buildbox-casd
# require specifying the storage path even for invoking the --version option.
#
- casd_args = [utils.get_host_tool("buildbox-casd")]
+ casd_args = [self.__buildbox_casd()]
casd_args.append("--version")
casd_args.append("/")
diff --git a/src/buildstream/_site.py b/src/buildstream/_site.py
index acc7fae1b..62dd7b583 100644
--- a/src/buildstream/_site.py
+++ b/src/buildstream/_site.py
@@ -43,3 +43,6 @@ build_all_template = os.path.join(root, "data", "build-all.sh.in")
# Module building script template
build_module_template = os.path.join(root, "data", "build-module.sh.in")
+
+# The bundled subprojects directory
+subprojects = os.path.join(root, "subprojects")
diff --git a/src/buildstream/_testing/_utils/site.py b/src/buildstream/_testing/_utils/site.py
index 3ce922365..d0e4cdbe0 100644
--- a/src/buildstream/_testing/_utils/site.py
+++ b/src/buildstream/_testing/_utils/site.py
@@ -55,7 +55,7 @@ try:
except ProgramNotFoundError:
HAVE_LZIP = False
-casd_path = utils.get_host_tool("buildbox-casd")
+casd_path = utils._get_host_tool_internal("buildbox-casd", search_subprojects_dir="buildbox")
CASD_SEPARATE_USER = bool(os.stat(casd_path).st_mode & stat.S_ISUID)
del casd_path
@@ -68,7 +68,7 @@ HAVE_SANDBOX = None
BUILDBOX_RUN = None
try:
- path = utils.get_host_tool("buildbox-run")
+ path = utils._get_host_tool_internal("buildbox-run", search_subprojects_dir="buildbox")
subprocess.run([path, "--capabilities"], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
BUILDBOX_RUN = os.path.basename(os.readlink(path))
HAVE_SANDBOX = "buildbox-run"
diff --git a/src/buildstream/sandbox/_sandboxbuildboxrun.py b/src/buildstream/sandbox/_sandboxbuildboxrun.py
index 765caf166..058eedaaa 100644
--- a/src/buildstream/sandbox/_sandboxbuildboxrun.py
+++ b/src/buildstream/sandbox/_sandboxbuildboxrun.py
@@ -34,10 +34,14 @@ from ._sandboxreapi import SandboxREAPI
# BuildBox-based sandbox implementation.
#
class SandboxBuildBoxRun(SandboxREAPI):
+ @classmethod
+ def __buildbox_run(cls):
+ return utils._get_host_tool_internal("buildbox-run", search_subprojects_dir="buildbox")
+
@classmethod
def check_available(cls):
try:
- path = utils.get_host_tool("buildbox-run")
+ path = cls.__buildbox_run()
except utils.ProgramNotFoundError as Error:
cls._dummy_reasons += ["buildbox-run not found"]
raise SandboxError(" and ".join(cls._dummy_reasons), reason="unavailable-local-sandbox") from Error
@@ -92,7 +96,7 @@ class SandboxBuildBoxRun(SandboxREAPI):
action_file.flush()
buildbox_command = [
- utils.get_host_tool("buildbox-run"),
+ self.__buildbox_run(),
"--remote={}".format(casd_process_manager._connection_string),
"--action={}".format(action_file.name),
"--action-result={}".format(result_file.name),
diff --git a/src/buildstream/utils.py b/src/buildstream/utils.py
index 0ea155d58..26f7925d3 100644
--- a/src/buildstream/utils.py
+++ b/src/buildstream/utils.py
@@ -54,6 +54,7 @@ from . import _signals
from ._exceptions import BstError
from .exceptions import ErrorDomain
from ._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
+from . import _site
# Contains utils that have been rewritten in Cython for speed benefits
# This makes them available when importing from utils
@@ -575,7 +576,9 @@ def link_files(
return result
-def get_host_tool(name: str) -> str:
+def get_host_tool(
+ name: str,
+) -> str:
"""Get the full path of a host tool
Args:
@@ -587,13 +590,7 @@ def get_host_tool(name: str) -> str:
Raises:
:class:`.ProgramNotFoundError`
"""
- search_path = os.environ.get("PATH")
- program_path = shutil.which(name, path=search_path)
-
- if not program_path:
- raise ProgramNotFoundError("Did not find '{}' in PATH: {}".format(name, search_path))
-
- return program_path
+ return _get_host_tool_internal(name)
def get_bst_version() -> Tuple[int, int]:
@@ -749,6 +746,35 @@ def get_umask():
return _UMASK
+# _get_host_tool_internal():
+#
+# Get the full path of a host tool, including tools bundled inside the Python package.
+#
+# Args:
+# name (str): The name of the program to search for
+# search_subprojects_dir (str): Optionally search in bundled subprojects directory
+#
+# Returns:
+# The full path to the program, if found
+#
+# Raises:
+# :class:`.ProgramNotFoundError`
+def _get_host_tool_internal(
+ name: str,
+ search_subprojects_dir: Optional[str] = None,
+) -> str:
+ search_path = os.environ.get("PATH", "").split(os.pathsep)
+ if search_subprojects_dir:
+ search_path.insert(0, os.path.join(_site.subprojects, search_subprojects_dir))
+
+ program_path = shutil.which(name, path=os.pathsep.join(search_path))
+
+ if not program_path:
+ raise ProgramNotFoundError("Did not find '{}' in PATH: {}".format(name, search_path))
+
+ return program_path
+
+
# _get_dir_size():
#
# Get the disk usage of a given directory in bytes.
diff --git a/tests/integration/cachedfail.py b/tests/integration/cachedfail.py
index da1fff6c2..2700f6e11 100644
--- a/tests/integration/cachedfail.py
+++ b/tests/integration/cachedfail.py
@@ -223,7 +223,7 @@ def test_host_tools_errors_are_not_cached(cli, datafiles, tmp_path):
# Create symlink to buildbox-casd to work with custom PATH
buildbox_casd = tmp_path.joinpath("bin/buildbox-casd")
buildbox_casd.parent.mkdir()
- os.symlink(utils.get_host_tool("buildbox-casd"), str(buildbox_casd))
+ os.symlink(utils._get_host_tool_internal("buildbox-casd", search_subprojects_dir="buildbox"), str(buildbox_casd))
project = str(datafiles)
element_path = os.path.join(project, "elements", "element.bst")
diff --git a/tests/sandboxes/selection.py b/tests/sandboxes/selection.py
index 403ff1f88..efb22dcc4 100644
--- a/tests/sandboxes/selection.py
+++ b/tests/sandboxes/selection.py
@@ -33,7 +33,7 @@ def test_dummy_sandbox_fallback(cli, datafiles, tmp_path):
# Create symlink to buildbox-casd to work with custom PATH
buildbox_casd = tmp_path.joinpath("bin/buildbox-casd")
buildbox_casd.parent.mkdir()
- os.symlink(utils.get_host_tool("buildbox-casd"), str(buildbox_casd))
+ os.symlink(utils._get_host_tool_internal("buildbox-casd", search_subprojects_dir="buildbox"), str(buildbox_casd))
project = str(datafiles)
element_path = os.path.join(project, "elements", "element.bst")
[buildstream] 03/06: Build and release wheels packages as part of GitHub Actions
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit a24be10eeb0aa882702d8a09ae6ee44b1a8661d2
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Tue Aug 16 18:04:31 2022 +0200
Build and release wheels packages as part of GitHub Actions
This updates the CI config to:
* build wheel packages in pre-merge CI
* download BuildBox release binaries to build the wheels
* these can be downloaded as action artifacts
* test built wheel packages in pre-merge CI
* build sdist and wheel packages on each release tag, and upload
them to Test PyPI at https://test.pypi.org/project/BuildStream/
The new workflows are based on examples at
https://cibuildwheel.readthedocs.io/en/stable/setup/, avoiding use of
the GitHub Actions from `pypa/*` as these are not permitted to be used
in apache/buildstream project.
---
.github/compose/ci.docker-compose.yml | 24 +++++
.../wheel-helpers/fetch-latest-buildbox-release.sh | 16 +++
.github/wheel-helpers/test-wheel-manylinux.sh | 20 ++++
.github/workflows/ci.yml | 54 +++++++++-
.github/workflows/merge.yml | 2 +-
.github/workflows/release.yml | 111 ++++++++++++++++++++-
6 files changed, 222 insertions(+), 5 deletions(-)
diff --git a/.github/compose/ci.docker-compose.yml b/.github/compose/ci.docker-compose.yml
index 83677af1e..f616f1a7b 100644
--- a/.github/compose/ci.docker-compose.yml
+++ b/.github/compose/ci.docker-compose.yml
@@ -85,3 +85,27 @@ services:
mypy:
<<: *tests-template
command: tox -e mypy
+
+ # Test that each BuildStream+BuildBox wheel package can install and run.
+ # on the PyPA official 'manylinux' images that define the base ABI for
+ # Python binary packages.
+
+ wheels-manylinux_2_28-cp37:
+ <<: *tests-template
+ image: quay.io/pypa/manylinux_2_28_x86_64
+ command: .github/wheel-helpers/test-wheel-manylinux.sh cp37-cp37m-manylinux_2_28_x86_64 /opt/python/cp37-cp37m/bin/python3
+
+ wheels-manylinux_2_28-cp38:
+ <<: *tests-template
+ image: quay.io/pypa/manylinux_2_28_x86_64
+ command: .github/wheel-helpers/test-wheel-manylinux.sh cp38-cp38-manylinux_2_28_x86_64 /opt/python/cp38-cp38/bin/python3
+
+ wheels-manylinux_2_28-cp39:
+ <<: *tests-template
+ image: quay.io/pypa/manylinux_2_28_x86_64
+ command: .github/wheel-helpers/test-wheel-manylinux.sh cp39-cp39-manylinux_2_28_x86_64 /opt/python/cp39-cp39/bin/python3
+
+ wheels-manylinux_2_28-cp310:
+ <<: *tests-template
+ image: quay.io/pypa/manylinux_2_28_x86_64
+ command: .github/wheel-helpers/test-wheel-manylinux.sh cp310-cp310-manylinux_2_28_x86_64 /opt/python/cp310-cp310/bin/python3
diff --git a/.github/wheel-helpers/fetch-latest-buildbox-release.sh b/.github/wheel-helpers/fetch-latest-buildbox-release.sh
new file mode 100755
index 000000000..0555efd49
--- /dev/null
+++ b/.github/wheel-helpers/fetch-latest-buildbox-release.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# Download latest release binaries of BuildBox. These are statically linked
+# binaries produced by the buildbox-integration GitLab project, which we
+# bundle into BuildStream wheel packages.
+
+set -eux
+
+curl -O https://gitlab.com/buildgrid/buildbox/buildbox-integration/-/releases/permalink/latest/downloads/binaries.tgz
+
+mkdir -p src/buildstream/subprojects/buildbox
+tar --extract --file ./binaries.tgz --directory src/buildstream/subprojects/buildbox
+
+cd src/buildstream/subprojects/buildbox
+rm buildbox-run
+mv buildbox-run-bubblewrap buildbox-run
diff --git a/.github/wheel-helpers/test-wheel-manylinux.sh b/.github/wheel-helpers/test-wheel-manylinux.sh
new file mode 100755
index 000000000..a780ab412
--- /dev/null
+++ b/.github/wheel-helpers/test-wheel-manylinux.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+# Script to test that a generated BuildStream+BuildBox wheel package is
+# functional in the PyPA "manylinux" container images.
+#
+# The test is run via `run-ci.sh` which in turn uses `docker-compose` to
+# execute this script.
+
+set -eux
+
+COMPATIBILITY_TAGS=$1
+PYTHON=$2
+
+dnf install -y bubblewrap
+
+"$PYTHON" -m venv /tmp/venv
+/tmp/venv/bin/pip3 install ./wheelhouse/BuildStream-*-$COMPATIBILITY_TAGS.whl buildstream-plugins
+
+cd doc/examples/autotools
+/tmp/venv/bin/bst build hello.bst
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c3b3e4bfc..81d143576 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -83,7 +83,7 @@ jobs:
run: |
${GITHUB_WORKSPACE}/.github/run-ci.sh --service ${{ matrix.test-name }}
- docs:
+ build_docs:
runs-on: ubuntu-18.04
steps:
- name: Check out repository
@@ -108,3 +108,55 @@ jobs:
with:
name: docs
path: doc/build/html
+
+ build_wheels:
+ name: Build Python wheel packages on ${{ matrix.os }}
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os: [ubuntu-20.04]
+
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - name: Fetch latest BuildBox release
+ run: ${GITHUB_WORKSPACE}/.github/wheel-helpers/fetch-latest-buildbox-release.sh
+
+ - name: Build wheels
+ run: pipx run cibuildwheel==2.8.1
+
+ - uses: actions/upload-artifact@v3
+ with:
+ name: wheels
+ path: ./wheelhouse/*.whl
+
+ test_wheels:
+ name: "Test Python packages: ${{ matrix.test-name }}"
+ needs: [build_wheels]
+ runs-on: ubuntu-20.04
+
+ strategy:
+ matrix:
+ # The names here should map to a valid service defined in
+ # "../compose/ci.docker-compose.yml"
+ test-name:
+ - wheels-manylinux_2_28-cp37
+ - wheels-manylinux_2_28-cp38
+ - wheels-manylinux_2_28-cp39
+ - wheels-manylinux_2_28-cp310
+
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - uses: actions/download-artifact@v3
+ with:
+ name: wheels
+ path: ./wheelhouse
+
+ - name: Run tests with Docker Compose
+ run: |
+ ${GITHUB_WORKSPACE}/.github/run-ci.sh ${{ matrix.test-name }}
diff --git a/.github/workflows/merge.yml b/.github/workflows/merge.yml
index b9c526375..d9bf499e2 100644
--- a/.github/workflows/merge.yml
+++ b/.github/workflows/merge.yml
@@ -7,7 +7,7 @@ on:
jobs:
build_docs:
- name: Build documentation
+ name: "Build documentation tarball"
runs-on: ubuntu-18.04
steps:
- name: Checkout code
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 36f54c3ed..ccd0f717c 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -1,4 +1,4 @@
-name: Upload Release Asset
+name: Release actions
on:
push:
@@ -6,8 +6,8 @@ on:
- '*.*.*'
jobs:
- build:
- name: Upload Release Asset
+ build_docs:
+ name: "Build documentation tarball"
runs-on: ubuntu-18.04
steps:
- name: Checkout code
@@ -32,9 +32,114 @@ jobs:
tar -C doc/build/html -zcf docs.tgz .
+ - uses: actions/upload-artifact@v3
+ with:
+ name: docs
+ path: docs.tgz
+
+ build_sdist:
+ name: "Build Python source distribution tarball"
+ runs-on: ubuntu-20.04
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - name: Build sdist
+ run: pipx run build --sdist
+
+ - uses: actions/upload-artifact@v3
+ with:
+ name: sdist
+ path: dist/*.tar.gz
+
+ build_wheels:
+ name: Build Python wheel packages on ${{ matrix.os }}
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os: [ubuntu-20.04]
+
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - name: Fetch latest BuildBox release
+ run: ${GITHUB_WORKSPACE}/.github/wheel-helpers/fetch-latest-buildbox-release.sh
+
+ - name: Build wheels
+ run: pipx run cibuildwheel==2.8.1
+
+ - uses: actions/upload-artifact@v3
+ with:
+ name: wheels
+ path: ./wheelhouse/*.whl
+
+ test_wheels:
+ name: Test Python wheel packages on ${{ matrix.os }}
+ needs: [build_wheels]
+ runs-on: ubuntu-20.04
+
+ strategy:
+ matrix:
+ # The names here should map to a valid service defined in
+ # "../compose/ci.docker-compose.yml"
+ test-name:
+ - wheels-manylinux_2_28-cp37
+ - wheels-manylinux_2_28-cp38
+ - wheels-manylinux_2_28-cp39
+ - wheels-manylinux_2_28-cp310
+
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - uses: actions/download-artifact@v3
+ with:
+ name: wheels
+ path: ./wheelhouse
+
+ - name: Run tests with Docker Compose
+ run: |
+ ${GITHUB_WORKSPACE}/.github/run-ci.sh ${{ matrix.test-name }}
+
+ upload_github_release:
+ name: Upload GitHub release assets
+ needs: [build_docs, build_sdist, build_wheels, test_wheels]
+ runs-on: ubuntu-20.04
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ fetch-depth: 0
+
+ - uses: actions/download-artifact@v3
+ with:
+ name: docs
+
- name: Upload release assets
run: |
tag_name="${GITHUB_REF##*/}"
hub release create -a "docs.tgz" -m "$tag_name" "$tag_name"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+ upload_pypi_release:
+ name: Upload PyPI release assets
+ needs: [build_docs, build_sdist, build_wheels, test_wheels]
+ runs-on: ubuntu-20.04
+ steps:
+ - uses: actions/download-artifact@v3
+ with:
+ name: sdist
+ path: dist
+
+ - uses: actions/download-artifact@v3
+ with:
+ name: wheels
+ path: dist
+
+ - name: Upload to PyPI
+ run: |
+ pipx run twine upload --repository pypi --username __token__ --password "${{ secrets.PYPI_TOKEN }}" dist/*
[buildstream] 02/06: .github/workflows/merge.yml: Rename 'build' and 'publish'
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch tristan/build-wheels
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 0d31531ce5a4d0aa42776bae1e1bec7a3cb88fac
Author: Sam Thursfield <sa...@codethink.co.uk>
AuthorDate: Tue Aug 16 18:02:45 2022 +0200
.github/workflows/merge.yml: Rename 'build' and 'publish'
We are going to build and publish more than just docs, so add _docs
to these job names to make it clear.
---
.github/workflows/merge.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/merge.yml b/.github/workflows/merge.yml
index e221ce4bf..b9c526375 100644
--- a/.github/workflows/merge.yml
+++ b/.github/workflows/merge.yml
@@ -6,7 +6,7 @@ on:
- master
jobs:
- build:
+ build_docs:
name: Build documentation
runs-on: ubuntu-18.04
steps:
@@ -42,7 +42,7 @@ jobs:
doc/build/html
docs.tgz
- publish:
+ publish_docs:
needs: build
runs-on: ubuntu-18.04
steps: