You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2021/02/04 07:17:02 UTC
[buildstream] branch juerg/re-debug created (now e86a4ab)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch juerg/re-debug
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at e86a4ab disable external tests
This branch includes the following new commits:
new a54dcf0 _sandboxremote.py: debug
new d0fb8fc CI debug
new d5820d0 Revert "buildgrid-compose.yml: reduce log level to WARN"
new e86a4ab disable external tests
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 01/04: _sandboxremote.py: debug
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch juerg/re-debug
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit a54dcf06c5dcab699451ec9b07721bb0c4e414de
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Dec 19 10:11:00 2019 +0100
_sandboxremote.py: debug
---
src/buildstream/sandbox/_sandboxremote.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/buildstream/sandbox/_sandboxremote.py b/src/buildstream/sandbox/_sandboxremote.py
index 5ec1c97..0789bf5 100644
--- a/src/buildstream/sandbox/_sandboxremote.py
+++ b/src/buildstream/sandbox/_sandboxremote.py
@@ -430,6 +430,10 @@ class SandboxRemote(SandboxREAPI):
if execution_response.status.message:
raise SandboxError(execution_response.status.message)
# Otherwise, report the failure in a more general manner
- raise SandboxError("Remote server failed at executing the build request.")
+ raise SandboxError(
+ "Remote server failed at executing the build request. gRPC status code: {}".format(
+ execution_response.status.code
+ )
+ )
return execution_response.result
[buildstream] 04/04: disable external tests
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch juerg/re-debug
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit e86a4abc7afa63935e44ce436560bcc83de37425
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Dec 19 10:51:54 2019 +0100
disable external tests
---
tox.ini | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tox.ini b/tox.ini
index 62737b0..834b7a1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -20,12 +20,12 @@ commands =
# Running with coverage reporting enabled
py{35,36,37}-!external-!nocover: pytest --basetemp {envtmpdir} --cov=buildstream --cov-config .coveragerc {posargs}
py{35,36,37}-!nocover: mkdir -p .coverage-reports
- external-!nocover: {envpython} {toxinidir}/tests/external_plugins.py {envtmpdir} --cov=buildstream --cov-append {posargs}
+ # external-!nocover: {envpython} {toxinidir}/tests/external_plugins.py {envtmpdir} --cov=buildstream --cov-append {posargs}
py{35,36,37}-!external-!nocover: mv {envtmpdir}/.coverage {toxinidir}/.coverage-reports/.coverage.{env:COVERAGE_PREFIX:}{envname}
py{35,36,37}-external-!nocover: - mv {envtmpdir}/.coverage {toxinidir}/.coverage-reports/.coverage.{env:COVERAGE_PREFIX:}{envname}
# Running with coverage reporting disabled
py{35,36,37}-!external-nocover: pytest --basetemp {envtmpdir} {posargs}
- external-nocover: {envpython} {toxinidir}/tests/external_plugins.py {envtmpdir} {posargs}
+ # external-nocover: {envpython} {toxinidir}/tests/external_plugins.py {envtmpdir} {posargs}
deps =
py{35,36,37}: -rrequirements/requirements.txt
py{35,36,37}: -rrequirements/dev-requirements.txt
[buildstream] 02/04: CI debug
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch juerg/re-debug
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d0fb8fc938279cfdb02684b88e7e94b4ada2505f
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Dec 19 10:11:36 2019 +0100
CI debug
---
.gitlab-ci.yml | 410 ---------------------------------------------------------
1 file changed, 410 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1fe2581..69195c1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,8 +1,3 @@
-include:
- - template: Code-Quality.gitlab-ci.yml
- - template: License-Management.gitlab-ci.yml
- - template: SAST.gitlab-ci.yml
-
image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:10-${DOCKER_IMAGE_VERSION}
cache:
@@ -64,133 +59,6 @@ variables:
paths:
- .coverage-reports
-tests-debian-9:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:9-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
-tests-debian-10:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:10-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
-tests-fedora-30:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:30-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
-tests-fedora-31:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
-tests-ubuntu-18.04:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-ubuntu:18.04-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
-tests-centos-7.6:
- <<: *tests
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-centos:7.6.1810-${DOCKER_IMAGE_VERSION}
-
-overnight-fedora-30-aarch64:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:aarch64-30-${DOCKER_IMAGE_VERSION}
- tags:
- - aarch64
- <<: *tests
- # We need to override the exclusion from the template
- # in order to run on schedules
- except: []
- only:
- - schedules
-
-tests-unix:
- # Use fedora here, to a) run a test on fedora and b) ensure that we
- # can get rid of ostree - this is not possible with debian-8
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
- variables:
- BST_FORCE_SANDBOX: "chroot"
-
- script:
-
- # We remove the Bubblewrap and OSTree packages here so that we catch any
- # codepaths that try to use them. Removing OSTree causes fuse-libs to
- # disappear unless we mark it as user-installed.
- - dnf mark install fuse-libs systemd-udev
- - dnf erase -y bubblewrap ostree
-
- # Since the unix platform is required to run as root, no user change required
- - ${TEST_COMMAND}
-
-tests-buildbox-run:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
- variables:
- BST_FORCE_SANDBOX: "buildbox-run"
-
-tests-userchroot:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
- variables:
- BST_FORCE_SANDBOX: "buildbox-run"
- BST_CAS_STAGING_ROOT: "/builds/userchroot"
-
- script:
- - mkdir -p "${INTEGRATION_CACHE}"
- - useradd -Um buildstream
-
- # Use buildbox-run-userchroot and hardlinking
- - ln -svf buildbox-run-userchroot /usr/local/bin/buildbox-run
- - rm -vf /usr/local/bin/buildbox-fuse
-
- # When using userchroot, buildbox-casd must run as a separate user
- - useradd -g buildstream buildbox-casd
- - chown buildbox-casd:buildstream /usr/local/bin/buildbox-casd
- - chmod u+s /usr/local/bin/buildbox-casd
-
- # Set up staging root with permissions required by userchroot,
- # must be on same filesystem as current directory to support hardlinks
- - mkdir -p "${BST_CAS_STAGING_ROOT}"
- - chown -R buildbox-casd:buildstream "${BST_CAS_STAGING_ROOT}"
- # userchroot doesn't allow group/world-writable base directory
- - chmod go-w /builds
- - echo buildbox-casd:${BST_CAS_STAGING_ROOT} > /etc/userchroot.conf
-
- - chown -R buildstream:buildstream .
-
- # Run the tests as a simple user to test for permission issues
- - su buildstream -c "umask 002 && ${TEST_COMMAND}"
- - su buildstream -c "umask 002 && ${EXTERNAL_TESTS_COMMAND}"
-
-tests-fedora-missing-deps:
- # Ensure that tests behave nicely while missing bwrap and ostree
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
- script:
- # We remove the Bubblewrap and OSTree packages here so that we catch any
- # codepaths that try to use them. Removing OSTree causes fuse-libs to
- # disappear unless we mark it as user-installed.
- - dnf mark install fuse-libs systemd-udev
- - dnf erase -y bubblewrap ostree
-
- - useradd -Um buildstream
- - chown -R buildstream:buildstream .
-
- - ${TEST_COMMAND}
-
-tests-fedora-update-deps:
- # Check if the tests pass after updating requirements to their latest
- # allowed version.
- allow_failure: true
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
-
- script:
- - useradd -Um buildstream
- - chown -R buildstream:buildstream .
-
- - make --always-make --directory requirements
- - cat requirements/*.txt
-
- - su buildstream -c "${TEST_COMMAND}"
-
tests-remote-execution:
allow_failure: true
image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
@@ -214,281 +82,3 @@ tests-remote-execution:
REMOTE_EXECUTION_SERVICE: http://docker:50051
SOURCE_CACHE_SERVICE: http://docker:50052
PYTEST_ARGS: "--color=yes --remote-execution"
-
-tests-spawn-multiprocessing-start-method:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- <<: *tests
- variables:
- BST_FORCE_START_METHOD: "spawn"
- script:
- # FIXME: Until all the tests pass as normal, override which tests will run here.
- - mkdir -p "${INTEGRATION_CACHE}"
- - useradd -Um buildstream
- - chown -R buildstream:buildstream .
- - su buildstream -c "tox -- ${PYTEST_ARGS} tests/{artifactcache,cachekey,elements,format,frontend,internals,plugins,sourcecache}"
-
-# Run type checkers
-mypy:
- stage: test
-
- script:
- - tox -e mypy
- except:
- - schedules
-
-# Lint separately from testing
-lint:
- # We can't use the default debian:9 based image here since that comes with
- # Python 3.5, and Black requires Python >= 3.6.
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- stage: test
-
- before_script:
- # Diagnostics
- - python3 --version
-
- script:
- - tox -e format-check,lint
- except:
- - schedules
-
-# Catch regressions in native windows support
-tests-win32-master:
- stage: test
- variables:
- LC_ALL: C.UTF-8
- LANG: C.UTF-8
- tags:
- - win32
- script:
- - tox -e win32
- only:
- - master
-
-# Optional test to catch regressions in native windows support on non-master branches
-tests-win32-non-master:
- stage: test
- variables:
- LC_ALL: C.UTF-8
- LANG: C.UTF-8
- tags:
- - win32
- script:
- - tox -e win32
- except:
- - master
- when: manual
-
-tests-wsl-master:
- stage: test
- variables:
- LC_ALL: C.UTF-8
- LANG: C.UTF-8
- tags:
- - wsl
- before_script:
- - mount
- - df -h
- - PATH=/root/.local/bin:$PATH tox --version
- script:
- # Install static buildbox-casd binary
- - wget https://buildbox-casd-binaries.nyc3.cdn.digitaloceanspaces.com/buildbox-casd-x86_64-linux-20191104-598100dd.tar.xz
- - tar -C /root/.local/bin -xf buildbox-casd-x86_64-linux-20191104-598100dd.tar.xz
-
- - PATH=/root/.local/bin:$PATH ${TEST_COMMAND}
- only:
- - master
-
-tests-wsl-non-master:
- stage: test
- variables:
- LC_ALL: C.UTF-8
- LANG: C.UTF-8
- tags:
- - wsl
- before_script:
- - mount
- - df -h
- - PATH=/root/.local/bin:$PATH tox --version
- script:
- # Install static buildbox-casd binary
- - wget https://buildbox-casd-binaries.nyc3.cdn.digitaloceanspaces.com/buildbox-casd-x86_64-linux-20191104-598100dd.tar.xz
- - tar -C /root/.local/bin -xf buildbox-casd-x86_64-linux-20191104-598100dd.tar.xz
-
- - PATH=/root/.local/bin:$PATH ${TEST_COMMAND}
- when: manual
- except:
- - master
-
-# Automatically build documentation for every commit, we want to know
-# if building documentation fails even if we're not deploying it.
-docs:
- stage: test
- variables:
- BST_FORCE_SESSION_REBUILD: 1
- script:
- - env BST_SOURCE_CACHE="$(pwd)/cache/integration-cache/sources" tox -e docs
- - mv doc/build/html public
- except:
- - schedules
- artifacts:
- paths:
- - public/
-
-.overnight-tests: &overnight-tests-template
- stage: test
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-${DOCKER_IMAGE_VERSION}
- variables:
- BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-plugins-experimental.git
- BST_EXT_REF: 0.12.0-173-gbe5ac19#egg=bst_plugins_experimental[ostree,cargo]
- FD_SDK_REF: freedesktop-sdk-19.08.5-buildstream2-0-g22588e2f41acecbcfc555942eb3086296bc14c6c
- before_script:
- - |
- mkdir -p "${HOME}/.config"
- cat <<EOF >"${HOME}/.config/buildstream.conf"
- scheduler:
- fetchers: 2
- EOF
- - dnf install -y ostree python3-gobject-base
- - pip3 install -r requirements/requirements.txt -r requirements/plugin-requirements.txt
- - pip3 wheel --wheel-dir wheels/ --no-deps .
- - pip3 install --no-index wheels/*
- - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}
- - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
- - git -C freedesktop-sdk checkout ${FD_SDK_REF}
- artifacts:
- paths:
- - "${HOME}/.cache/buildstream/logs"
- only:
- - schedules
-
-overnight-tests:
- <<: *overnight-tests-template
- script:
- - make -C freedesktop-sdk
- tags:
- - overnight-tests
-
-overnight-tests-no-cache:
- <<: *overnight-tests-template
- script:
- - sed -i '/artifacts:/,+1 d' freedesktop-sdk/project.conf
- - make -C freedesktop-sdk
- tags:
- - overnight-tests
-
-# These tests might be a bit more flaky since they randomize the order
-# They will also take longer to run. Hence run them only nightly
-overnight-randomized:
- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:31-master-97748867
- <<: *tests
-
- script:
- - mkdir -p "${INTEGRATION_CACHE}"
- - useradd -Um buildstream
- - chown -R buildstream:buildstream .
-
- # Don't run tests multiprocessed here, the randomized order doesn't like that
- - su buildstream -c "tox -e py35-randomized,py36-randomized,py37-randomized -- --color=yes --integration"
- - su buildstream -c "tox -e py35-randomized,py36-randomized,py37-randomized-external -- --color=yes --integration"
-
- # We need to override the exclusion from the template in order to run on schedules
- except: []
- only:
- - schedules
-
-#####################################################
-# Post stage #
-#####################################################
-
-analysis:
- stage: post
- script:
- - |
- pip3 install radon
- mkdir analysis
-
- - |
- echo "Calculating Maintainability Index"
- radon mi -s -j src/buildstream > analysis/mi.json
- radon mi -s src/buildstream
-
- - |
- echo "Calculating Cyclomatic Complexity"
- radon cc -a -s -j src/buildstream > analysis/cc.json
- radon cc -a -s src/buildstream
-
- - |
- echo "Calculating Raw Metrics"
- radon raw -s -j src/buildstream > analysis/raw.json
- radon raw -s src/buildstream
-
- except:
- - schedules
- artifacts:
- paths:
- - analysis/
-
-# Collate coverage reports
-#
-coverage:
- stage: post
- coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
- script:
- - cp -a .coverage-reports/ ./coverage-sources
- - tox -e coverage
- - cp -a .coverage-reports/ ./coverage-report
- dependencies:
- - tests-buildbox-run
- - tests-centos-7.6
- - tests-debian-9
- - tests-debian-10
- - tests-fedora-30
- - tests-fedora-31
- - tests-fedora-missing-deps
- - tests-fedora-update-deps
- - tests-remote-execution
- - tests-ubuntu-18.04
- - tests-unix
- - tests-userchroot
- except:
- - schedules
- artifacts:
- paths:
- - coverage-sources/
- - coverage-report/
-
-# Deploy, only for merges which land on master branch.
-#
-pages:
- stage: publish
- dependencies:
- - coverage
- - docs
- variables:
- ACME_DIR: public/.well-known/acme-challenge
- COVERAGE_DIR: public/coverage
- script:
- - mkdir -p ${ACME_DIR}
- # Required to finish the creation of the Let's Encrypt certificate,
- # which allows using https://docs.buildstream.build/ for accessing
- # the documentation.
- - echo ${ACME_CHALLENGE} > ${ACME_DIR}/$(echo ${ACME_CHALLENGE} | cut -c1-43)
- - mkdir -p ${COVERAGE_DIR}
- - cp -a ./coverage-report/ ${COVERAGE_DIR}
- artifacts:
- paths:
- - public/
- only:
- #
- # FIXME:
- #
- # Ideally we want to publish to a different subdir of
- # pages depending on which stable branch we are building here,
- # not currently automatically supported but can be worked around.
- #
- # See https://gitlab.com/gitlab-org/gitlab-ce/issues/35141
- #
- - master
- except:
- - schedules
[buildstream] 03/04: Revert "buildgrid-compose.yml: reduce log
level to WARN"
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch juerg/re-debug
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit d5820d0412fa79e0ecdb780d1d64cbe6596c1d16
Author: Jürg Billeter <j...@bitron.ch>
AuthorDate: Thu Dec 19 10:50:09 2019 +0100
Revert "buildgrid-compose.yml: reduce log level to WARN"
This reverts commit 9ae7b67979fe0863d3508943d5936ff88698b444.
---
.gitlab-ci/buildgrid-compose.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.gitlab-ci/buildgrid-compose.yml b/.gitlab-ci/buildgrid-compose.yml
index 95b98f7..0916581 100644
--- a/.gitlab-ci/buildgrid-compose.yml
+++ b/.gitlab-ci/buildgrid-compose.yml
@@ -28,7 +28,7 @@ services:
controller:
image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
command: [
- "bgd", "server", "start", "-v",
+ "bgd", "server", "start", "-vvv",
"/etc/buildgrid/default.conf"]
ports:
- 50051:50051
@@ -38,7 +38,7 @@ services:
bot:
image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
command: [
- "bgd", "bot", "--parent=", "-v",
+ "bgd", "bot", "--parent=", "-vvv",
"--remote=http://controller:50051",
"--remote-cas=http://controller:50051",
"buildbox",
@@ -57,7 +57,7 @@ services:
storage:
image: registry.gitlab.com/buildgrid/buildgrid.hub.docker.com/buildgrid:nightly
command: [
- "bgd", "server", "start", "-v",
+ "bgd", "server", "start", "-vvv",
"/etc/buildgrid/artifacts.conf"]
ports:
- 50052:50052