You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@buildstream.apache.org by tv...@apache.org on 2021/02/04 08:00:16 UTC
[buildstream] branch jmac/performance-ci created (now 7b71f50)
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a change to branch jmac/performance-ci
in repository https://gitbox.apache.org/repos/asf/buildstream.git.
at 7b71f50 CI.rst: New file: Documentation for continuous integration
This branch includes the following new commits:
new 08f6469 .gitlab-ci.yml: Add the performance test.
new 7b71f50 CI.rst: New file: Documentation for continuous integration
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[buildstream] 01/02: .gitlab-ci.yml: Add the performance test.
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch jmac/performance-ci
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 08f6469121932c9262ea9633e59242fef110feec
Author: Jim MacArthur <ji...@codethink.co.uk>
AuthorDate: Thu Mar 8 16:02:26 2018 +0000
.gitlab-ci.yml: Add the performance test.
This invokes the `benchmarks` tool to do simple performance tests
on the commit under test.
---
.gitlab-ci.yml | 52 +++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 49 insertions(+), 3 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a31e0fa..9813e15 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -15,9 +15,6 @@ before_script:
- mount
- df -h
- - useradd -Um buildstream
- - chown -R buildstream:buildstream .
-
#####################################################
# Prepare stage #
#####################################################
@@ -62,12 +59,20 @@ source_dist:
# Test stage #
#####################################################
+# This is a YAML anchor to set up users for linux-tests and unix-tests
+# (it will not work on linux-perf-tests).
+.user_setup: &shared_user_setup
+ before_script:
+ - useradd -Um buildstream;
+ - chown -R buildstream:buildstream .;
+
# Run premerge commits
#
linux-tests:
stage: test
variables:
PYTEST_ADDOPTS: "--color=yes"
+ <<: *shared_user_setup # Add the before_script from .user_setup
script:
- export INTEGRATION_CACHE="$(pwd)/cache/integration-cache"
@@ -99,6 +104,7 @@ unix-tests:
variables:
BST_FORCE_BACKEND: "unix"
PYTEST_ADDOPTS: "--color=yes"
+ <<: *shared_user_setup # Add the before_script from .user_setup
script:
- export INTEGRATION_CACHE="$(pwd)/cache/integration-cache"
@@ -149,6 +155,46 @@ docs:
- source_dist
+############################################
+# Performance test #
+############################################
+
+# This checks out a specific revision of the 'benchmark' repository and uses it to
+# run a benchmark on the version of BuildStream currently under test. It can only run
+# on specific runner hardware matching the 'benchmark' tag. This test is allowed to fail
+# at the moment.
+linux-perf-tests:
+ stage: test
+ allow_failure: true
+ tags:
+ - benchmarks
+ variables:
+ PYTEST_ADDOPTS: "--color=yes"
+ script:
+
+ # No need to install BuildStream here, just tell the benchmarks repo
+ # to use the commit SHA and repository.
+ - git clone https://gitlab.com/BuildStream/benchmarks.git
+ - cd benchmarks
+
+ # Use a fixed version of benchmarks so the results are comparable.
+ - git checkout 4e03d40d638eb2f3e2111d6eac9ec504d9d5737b
+ - |
+ cat > version_under_test.benchmark << EOF
+ version_defaults:
+ buildstream_repo: $CI_REPOSITORY_URL
+
+ versions:
+ - name: version-under-test
+ base_docker_ref: latest
+ buildstream_ref: master
+ buildstream_commit: $CI_COMMIT_SHA
+ EOF
+ - python3 -m bst_benchmarks -c configs/buildstream-ci.benchmark -c version_under_test.benchmark > results.json
+ artifacts:
+ paths:
+ - benchmarks/results.json
+
#####################################################
# Post stage #
#####################################################
[buildstream] 02/02: CI.rst: New file: Documentation for continuous
integration
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvb pushed a commit to branch jmac/performance-ci
in repository https://gitbox.apache.org/repos/asf/buildstream.git
commit 7b71f50312b53e90fd9f38a72a9a101a0e86e362
Author: Jim MacArthur <ji...@codethink.co.uk>
AuthorDate: Tue Mar 27 13:28:06 2018 +0100
CI.rst: New file: Documentation for continuous integration
---
CI.rst | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/CI.rst b/CI.rst
new file mode 100644
index 0000000..d32d152
--- /dev/null
+++ b/CI.rst
@@ -0,0 +1,30 @@
+Continuous integration and testing of BuildStream
+=================================================
+
+At the time of writing, BuildStream is hosted on GitLab's public website at https://gitlab.com/BuildStream/. It uses GitLab's continuous integration system to provide automated build and test of the main branch (`master`) and any branches pushed to the BuildStream repository.
+
+As a simplification, continuous testing consists of:
+
+* Creating a source distribution with `python3 setup.py sdist`
+* Extracting that distribution
+* Running `python3 setup.py test --addopts --integration`
+* Extracting the coverage results and marking them as a test artifact.
+
+These tests are run on in both Linux and Unix mode (Unix is not actually used to test Unix mode, but BST_FORCE_BACKEND: "unix" is used to test as much of that mode as we can).
+
+Performance tests are also run in Linux mode. Performance tests operate in a different manner, do not use the source distribution, and require a special runner to keep performance tests consistent. Performance tests are performed using the separate `benchmarks` repository from https://gitlab.com/BuildStream/benchmarks and the performance test procedure is defined by that repository, not by BuildStream.
+
+`.gitlab-ci.yml` contains the full details of the test procedure.
+
+
+Requirements
+============
+The project must be hosted on a GitLab instance for the `.gitlab-ci.yml` script to work.
+
+The public GitLab instance (https://gitlab.com/) has shared runners which are suitable for the functional tests. Shared runners must be enabled for the BuildStream project for this to work.
+
+To run the performance tests, one or more reference hardware systems must act as a runner for the BuildStream project. These runners should only process jobs with the tag 'benchmarks'. They should be set up to run one job at a time. They should be set to use the `shell` Executor.
+
+Instructions on setting up GitLab runners are provided by GitLab at https://docs.gitlab.com/runner/install/.
+
+Assuming BuildStream and the `benchmarks` repository are both hosted on GitLab, the performance runner machines should be enabled for *both* projects. This will enable you to run continuous testing for BuildStream and also test the benchmarking repository itself and compare results.