You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2021/02/04 08:24:55 UTC
[buildstream] branch chandan/test created (now 21b3366)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch chandan/test
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 21b3366 test
This branch includes the following new commits:
new 2bcf81d ci: Simplify pipelines using Docker Compose
new 7d72b35 ci: Enable parallelism for tests
new 27a3256 merge & release compose
new 21b3366 test
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 02/04: ci: Enable parallelism for tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch chandan/test
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7d72b3521ef1634e947cae87cc092829ab639c5b
Author: Chandan Singh <ch...@chandansingh.net>
AuthorDate: Tue Jan 12 23:15:49 2021 +0000
ci: Enable parallelism for tests
---
.github/compose/ci.docker-compose.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/compose/ci.docker-compose.yml b/.github/compose/ci.docker-compose.yml
index e2f420d..8d476a2 100644
--- a/.github/compose/ci.docker-compose.yml
+++ b/.github/compose/ci.docker-compose.yml
@@ -1,7 +1,7 @@
version: '3.4'
x-tests-template: &tests-template
- command: tox -vvvvv -- --color=yes --integration
+ command: tox -vvvvv -- --color=yes --integration -n 4
devices:
- /dev/fuse:/dev/fuse
environment:
[buildstream] 03/04: merge & release compose
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch chandan/test
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 27a32567d969d9de2a8355d8d50b29ba9f9bd21b
Author: Chandan Singh <ch...@chandansingh.net>
AuthorDate: Fri Jan 15 00:16:06 2021 +0000
merge & release compose
---
.github/workflows/merge.yml | 20 +++++++++-----------
.github/workflows/release.yml | 21 ++++++++++-----------
2 files changed, 19 insertions(+), 22 deletions(-)
diff --git a/.github/workflows/merge.yml b/.github/workflows/merge.yml
index 9facf26..9504154 100644
--- a/.github/workflows/merge.yml
+++ b/.github/workflows/merge.yml
@@ -1,8 +1,7 @@
name: Merge actions
env:
- CI_IMAGE: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:10-master-132813612
- BST_FORCE_SESSION_REBUILD: 1
+ CI_IMAGE_VERSION: master-241289109
on:
push:
@@ -20,16 +19,15 @@ jobs:
with:
fetch-depth: 0
- - name: Build documentation
+ - name: Give `testuser` ownership of the source directory
+ run: sudo chown -R 1000:1000 ${GITHUB_WORKSPACE}
+
+ - name: Build documentation using Docker Compose
run: |
- docker run \
- --privileged \
- --device /dev/fuse \
- --env BST_FORCE_SESSION_REBUILD \
- --volume /home/runner/work:/__w \
- --workdir /__w/buildstream/buildstream \
- $CI_IMAGE \
- tox -e docs
+ docker-compose \
+ --file ${GITHUB_WORKSPACE}/.github/compose/ci.docker-compose.yml \
+ run \
+ docs
# Include a tarball in the published docs, allowing for
# easy re-publishing of master docs on docs.buildstream.build
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 1153eab..722a0e8 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -1,8 +1,7 @@
name: Upload Release Asset
env:
- CI_IMAGE: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-master-177137613
- BST_FORCE_SESSION_REBUILD: 1
+ CI_IMAGE_VERSION: master-241289109
on:
push:
@@ -20,16 +19,16 @@ jobs:
with:
fetch-depth: 0
- - name: Build documentation
+ - name: Give `testuser` ownership of the source directory
+ run: sudo chown -R 1000:1000 ${GITHUB_WORKSPACE}
+
+ - name: Build documentation using Docker Compose
run: |
- docker run \
- --privileged \
- --device /dev/fuse \
- --env BST_FORCE_SESSION_REBUILD \
- --volume /home/runner/work:/__w \
- --workdir /__w/buildstream/buildstream \
- $CI_IMAGE \
- tox -e docs
+ docker-compose \
+ --file ${GITHUB_WORKSPACE}/.github/compose/ci.docker-compose.yml \
+ run \
+ docs
+
tar -C doc/build/html -zcf docs.tgz .
- name: Upload release assets
[buildstream] 01/04: ci: Simplify pipelines using Docker Compose
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch chandan/test
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 2bcf81d5685ed245f999568d7d5a772e262f2af7
Author: Chandan Singh <ch...@chandansingh.net>
AuthorDate: Tue Jan 12 21:27:57 2021 +0000
ci: Simplify pipelines using Docker Compose
Replace our bespoke and rather long to type `docker` CLI invocations
with more manageable docker-compose commands. This should also be useful
for better replicating what the CI does on laptops, and also to help
users to run tests inside a container (especially when that's not
possible natively like on MacOS).
On a technical note, Compose does support YAML anchors, unlike GitHub
Actions. That allows us to reuse the common configuration options
without having to duplicate them everywhere.
This also removes most of the duplication in the image names.
---
.github/compose/ci.docker-compose.yml | 67 +++++++++++++++
.github/workflows/ci.yml | 150 ++++++----------------------------
2 files changed, 94 insertions(+), 123 deletions(-)
diff --git a/.github/compose/ci.docker-compose.yml b/.github/compose/ci.docker-compose.yml
new file mode 100644
index 0000000..e2f420d
--- /dev/null
+++ b/.github/compose/ci.docker-compose.yml
@@ -0,0 +1,67 @@
+version: '3.4'
+
+x-tests-template: &tests-template
+ command: tox -vvvvv -- --color=yes --integration
+ devices:
+ - /dev/fuse:/dev/fuse
+ environment:
+ BST_PLUGINS_EXPERIMENTAL_VERSION: 1.93.4
+ TOXENV: py36,py37,py38-nocover,py36-plugins,py37-plugins,py38-plugins-nocover
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-${CI_IMAGE_VERSION:-latest}
+ privileged: true
+ volumes:
+ - ../..:/home/testuser/buildstream
+ working_dir: /home/testuser/buildstream
+
+
+services:
+
+ fedora-31:
+ <<: *tests-template
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${CI_IMAGE_VERSION:-latest}
+
+ fedora-32:
+ <<: *tests-template
+
+ debian-10:
+ <<: *tests-template
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:10-${CI_IMAGE_VERSION:-latest}
+
+ ubuntu-18.04:
+ <<: *tests-template
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-ubuntu:18.04-${CI_IMAGE_VERSION:-latest}
+
+ centos-7.7.1908:
+ <<: *tests-template
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-centos:7.7.1908-${CI_IMAGE_VERSION:-latest}
+
+ # Ensure that tests also pass in the absence of a sandboxing tool
+ fedora-missing-deps:
+ <<: *tests-template
+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:minimal-${CI_IMAGE_VERSION:-latest}
+
+ # Ensure that tests also pass without `--develop` flag
+ no-usedevelop:
+ <<: *tests-template
+ environment:
+ TOXENV: py36-nocover,py37-nocover,py38-nocover
+
+ # Test the master version of external plugins
+ plugins-master:
+ <<: *tests-template
+ environment:
+ BST_PLUGINS_EXPERIMENTAL_VERSION: master
+
+ docs:
+ <<: *tests-template
+ command: tox -e docs
+ environment:
+ BST_FORCE_SESSION_REBUILD: 1
+
+ lint:
+ <<: *tests-template
+ command: tox -e lint
+
+ mypy:
+ <<: *tests-template
+ command: tox -e mypy
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e9f35c9..c811486 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -26,7 +26,6 @@ on:
# - publishing docs to gh-pages
# - persistent artifact cache
# - overnight jobs
-# - other one-off jobs like missing-deps, plugin jobs etc
# - wsl tasks (TODO: Check if GitHub's Windows runners allow WSL)
#
# New opportunities:
@@ -34,44 +33,32 @@ on:
# - standardize WSL tasks by using GitHub-provided runners
env:
- CI_IMAGE: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-master-177137613
- PYTEST_ARGS: --color=yes --integration
+ CI_IMAGE_VERSION: master-241289109
jobs:
tests:
runs-on: ubuntu-20.04
continue-on-error: ${{ matrix.allow-failure || false }}
- env:
- CI_IMAGE_PREFIX: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite
- CI_IMAGE_SUFFIX: master-177137613
- TOXENV: py36,py37,py38-nocover,py36-plugins,py37-plugins,py38-plugins-nocover
- # FIXME: De-duplicate against tox.ini
- BST_PLUGINS_EXPERIMENTAL_VERSION: 1.93.4
-
strategy:
fail-fast: false
matrix:
- # Main test targets, the name defines the image that will be used as
- # the base for running tests.
+ # The names here should map to a valid service defined in
+ # "../compose/ci.docker-compose.yml"
test-name:
- - debian:10
- - fedora:31
- - fedora:32
- - ubuntu:18.04
- - centos:7.7.1908
+ - debian-10
+ - fedora-31
+ - fedora-32
+ - ubuntu-18.04
+ - centos-7.7.1908
+ - fedora-missing-deps
+ - no-usedevelop
+ - lint
+ - mypy
include:
- # Ensure that tests also pass without `--develop` flag.
- - test-name: no-usedevelop
- image-name: fedora:32
- toxenv: py36-nocover,py37-nocover,py38-nocover
-
- # Test the master version of some external plugins
- test-name: plugins-master
- image-name: fedora:32
- bst-plugins-experimental-version: master
allow-failure: true
steps:
@@ -80,121 +67,38 @@ jobs:
# BuildStream requires tags to be able to find its version.
with:
fetch-depth: 0
- # XXX: Our run command looks like a monstrosity because we need to
- # specify `--device /dev/fuse` and there's no way to do that using the
- # `container` directive directly.
- # This is also why we have forward environment variables by hand.
- # TODO: In future, we should find a way to simplify this. See some
- # relevant discussion at:
- # https://github.community/t/how-to-run-privileged-docker-container/16431.
- # XXX: Value of `volume` and `workdir` must match how GitHub
- # Actions sets up paths.
- # TODO: Have test user pre-created in the test image.
- - name: Run tox inside a container
- run: |
-
- cat << EOF > runtox.sh
- #!/bin/bash
-
- # Create user
- useradd -Um buildstream
- chown -R buildstream:buildstream .
- # Diagnostics
- echo "Running diagnostics checks"
- mount
- df -h
- tox --version
+ - name: Give `testuser` ownership of the source directory
+ run: sudo chown -R 1000:1000 ${GITHUB_WORKSPACE}
- # Run tox as user, ensure we have a login shell
- echo "Running tests"
- su buildstream -c '/bin/bash --login -c "tox -vvvvv -- $PYTEST_ARGS"'
- EOF
-
- chmod +x runtox.sh
-
- docker run \
- --privileged \
- --device /dev/fuse \
- --env PYTEST_ARGS \
- --env TOXENV=${{ matrix.toxenv || env.TOXENV }} \
- --env BST_PLUGINS_EXPERIMENTAL_VERSION=${{ matrix.bst-plugins-experimental-version || env.BST_PLUGINS_EXPERIMENTAL_VERSION }} \
- --volume /home/runner/work:/__w \
- --workdir /__w/buildstream/buildstream \
- "$CI_IMAGE_PREFIX"-${{ matrix.image-name || matrix.test-name }}-"$CI_IMAGE_SUFFIX" \
- ./runtox.sh
-
- tests-fedora-missing-deps:
- runs-on: ubuntu-20.04
- container: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-master-177137613
- steps:
- - name: Check out repository
- uses: actions/checkout@v2
- # BuildStream requires tags to be able to find its version.
- with:
- fetch-depth: 0
- - name: Uninstall Bubblewrap and OSTree
- # We remove the Bubblewrap and OSTree packages here so that we catch any
- # codepaths that try to use them. Removing OSTree causes fuse-libs to
- # disappear unless we mark it as user-installed.
+ - name: Run tests with Docker Compose
run: |
- dnf mark install fuse-libs systemd-udev
- dnf erase -y bubblewrap ostree
- - name: Add test user
- run: |
- useradd -Um buildstream
- chown -R buildstream:buildstream .
- - name: Run tests
- run: su buildstream -c "tox -- $PYTEST_ARGS"
+ docker-compose \
+ --file ${GITHUB_WORKSPACE}/.github/compose/ci.docker-compose.yml \
+ run \
+ ${{ matrix.test-name }}
- mypy:
+ docs:
runs-on: ubuntu-20.04
- container: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-master-177137613
steps:
- name: Check out repository
uses: actions/checkout@v2
# BuildStream requires tags to be able to find its version.
with:
fetch-depth: 0
- - name: Run tox inside a container
- run: tox -e mypy
- lint:
- runs-on: ubuntu-20.04
- container: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:32-master-177137613
- steps:
- - name: Check out repository
- uses: actions/checkout@v2
- # BuildStream requires tags to be able to find its version.
- with:
- fetch-depth: 0
- - name: Run tox inside a container
- run: tox -e format-check,lint
+ - name: Give `testuser` ownership of the source directory
+ run: sudo chown -R 1000:1000 ${GITHUB_WORKSPACE}
- docs:
- runs-on: ubuntu-20.04
- env:
- BST_FORCE_SESSION_REBUILD: 1
- steps:
- - name: Check out repository
- uses: actions/checkout@v2
- # BuildStream requires tags to be able to find its version.
- with:
- fetch-depth: 0
- - name: Run tox inside a container
+ - name: Build documentation using Docker Compose
run: |
- docker run \
- --privileged \
- --device /dev/fuse \
- --env BST_FORCE_SESSION_REBUILD \
- --volume /home/runner/work:/__w \
- --workdir /__w/buildstream/buildstream \
- $CI_IMAGE \
- tox -e docs
+ docker-compose \
+ --file ${GITHUB_WORKSPACE}/.github/compose/ci.docker-compose.yml \
+ run \
+ docs
- name: Upload artifacts
uses: actions/upload-artifact@v2
with:
name: docs
path: doc/build/html
-
[buildstream] 04/04: test
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch chandan/test
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 21b3366223e5139c37bfe9642b045f2d348b077f
Author: Chandan Singh <ch...@chandansingh.net>
AuthorDate: Fri Jan 15 00:16:23 2021 +0000
test
---
.github/workflows/ci.yml | 49 +++++++-----------------------------------------
1 file changed, 7 insertions(+), 42 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c811486..6dea76b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -36,48 +36,6 @@ env:
CI_IMAGE_VERSION: master-241289109
jobs:
- tests:
- runs-on: ubuntu-20.04
- continue-on-error: ${{ matrix.allow-failure || false }}
-
- strategy:
- fail-fast: false
- matrix:
-
- # The names here should map to a valid service defined in
- # "../compose/ci.docker-compose.yml"
- test-name:
- - debian-10
- - fedora-31
- - fedora-32
- - ubuntu-18.04
- - centos-7.7.1908
- - fedora-missing-deps
- - no-usedevelop
- - lint
- - mypy
-
- include:
- - test-name: plugins-master
- allow-failure: true
-
- steps:
- - name: Check out repository
- uses: actions/checkout@v2
- # BuildStream requires tags to be able to find its version.
- with:
- fetch-depth: 0
-
- - name: Give `testuser` ownership of the source directory
- run: sudo chown -R 1000:1000 ${GITHUB_WORKSPACE}
-
- - name: Run tests with Docker Compose
- run: |
- docker-compose \
- --file ${GITHUB_WORKSPACE}/.github/compose/ci.docker-compose.yml \
- run \
- ${{ matrix.test-name }}
-
docs:
runs-on: ubuntu-20.04
steps:
@@ -97,6 +55,13 @@ jobs:
run \
docs
+ # Restore permissions to the current user
+ sudo chown -R ${USER} ${GITHUB_WORKSPACE}
+
+ # Include a tarball in the published docs, allowing for
+ # easy re-publishing of master docs on docs.buildstream.build
+ tar -C doc/build/html -zcf docs.tgz .
+
- name: Upload artifacts
uses: actions/upload-artifact@v2
with: