You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by ko...@apache.org on 2022/01/18 16:29:18 UTC
[couchdb] branch jenkins-dynamic-matrix updated (cebe188 -> 400f2a4)
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a change to branch jenkins-dynamic-matrix
in repository https://gitbox.apache.org/repos/asf/couchdb.git.
discard cebe188 Use native Map.get(key, default) for node label
discard 57949ce Need to escape for arch to flow through to shell
discard 9ddcdb4 Remove use of withEnv
discard 2b2833d Provide some extra comments
discard 636869a Parameterize minimum allowed Erlang version
discard e73f4ed No need for indirection
discard e6e9984 No need for indirection because these are only invoked once
discard f270a7c Fix make_packages
discard 5bd3cfa Make sure to actually fail the build
discard 499bf14 Fix tar invocation, use relative paths in package scripts
discard 918f8e7 Last refactor for today
discard c6f53cf Fix failFast config for scripted pipeline
discard e487dcd Fix platform -> PLATFORM env var
discard 0cbb71c just to be safe
discard 0fab3b7 Use gmake for build-report, streamline debugging
discard 842ffd4 Allow customizing docker agent by label
discard c3b6809 Fix stage name
discard 6e11dff Try to get back to good working state
discard 7aa914a See if try/catch/finally is causing UX troubles
discard 891f3b0 More experiments
discard 4676d66 Try this to name the branch more succinctly
discard 01b3000 Fixes
discard bd1c0e0 Improve workspace cleanup practices
discard 5963546 Final cleanup for the night
discard 81435f8 Use try-catch-finally to recover post function
omit a4e1ae8 Bring back into env
omit 7dcee89 Remove disallowed block
omit 6fa8296 Attempt a simplified container stage
omit 78cbc25 Why would this need to be quoted?
omit afac5c9 Ensure survives
omit f192ef5 More elimination of declarative bits
omit 57eb300 Try to get native builds working first
omit bd90a55 Is a stage even required?
omit ceaf237 Allow any agent
omit fa798ff Dynamically generate parallel stages
omit d76c488 Ensure the failure report is generated in the right directory
omit b6a75fb Add labels to improve UX
omit 8400498 Attempt to add freebsd back to the mix
omit 21ebc36 Don't update deps when building from tarball
omit 3aa91a0 Fix the cleaner
omit 7420458 Skip slow erlfmt-check for now
omit 18c486e Improve visibility of release generation steps
omit cc49869 Fix PATH in non-container builds
omit c289e31 Use new createrepo
omit 6c51421 Stages need to be string literals
omit e6e97a9 More refinements and alignment across all platforms
omit 3818ca0 Once more into the breaach
omit c7d4c25 Debug cwd issues
omit cac2cc7 Revert "Can a matrix exist alongside another stage?"
omit 03b9a49 Can a matrix exist alongside another stage?
omit 2a9758d Break build/test stage into multiple steps
omit 9b13780 Tweak Elixir formatting
omit d8ec48a Use beforeAgent to avoid blocking on non-existent agent
omit 4c67898 macOS uses SM60
omit 9fcf942 Temporarily drop FreeBSD
omit 6ef86b0 Generate tarball with old Erlang
omit 7825197 Reorder declarations
omit b94e39f Fix stages nesting
omit 02830dd Try spidermonkey version as an axis
omit e0ca5f7 Combine all into one matrix
omit e97f0c7 Experiment with DRY matrix build
omit d9226e8 Some fixes
omit b68e22d Bump Erlang and Debian versions in CI
omit c97f8d6 Remove CI support for Ubuntu 16.04
add 99b6871 Remove CI support for Ubuntu 16.04
new 040f03b Tweak Elixir formatting
new 400f2a4 Refactor build to dynamically generate test stages
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cebe188)
\
N -- N -- N refs/heads/jenkins-dynamic-matrix (400f2a4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build-aux/Jenkinsfile.full | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
[couchdb] 02/02: Refactor build to dynamically generate test stages
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 400f2a42597857107093da54dd69a4ce9bdd82dd
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Sat Jan 15 09:56:31 2022 -0500
Refactor build to dynamically generate test stages
This is one of those situations where you go in to make a small change,
see an opportunity for some refactoring, and get sucked into a rabbit
hole that leaves you wondering if you have any idea how computers
actually work. My initial goal was simply to update the Erlang version
used in our binary packages to a modern supported release. Along the
way I decided I wanted to figure out how to eliminate all the copypasta
we generate for making any change to this file, and after a few days of
hacking here we are. This rewrite has the following features:
* Updates to use Debian 11 (current stable) as the base image for
building releases and packaging repos.
* Defaults to Erlang 24.2 as the embedded Erlang version in packages.
* Dynamically generates the parallel build stages used to test and
package CouchDB on various OSes. This is accomplished through a bit
of scripted pipeline code that relies on two new methods defined at
the beginning of the Jenkinsfile, one for "native" builds on macOS
and FreeBSD and one for container-based builds. See comments in the
Jenkinsfile for additional details.
* Expands commands like `make check` into a series of steps to improve
visibility. The Jenkins UI will now show the time spent in each step
of the build process, and if a step (e.g. `make eunit`) fails it will
only expand the logs for that step by default instead of showing the
logs for the entire build stage. The downside is that if we do make
changes to the series of targets underneath `check` we need to
remember to update the Jenkinsfile as well.
---
build-aux/Jenkinsfile.full | 754 +++++++++++++++------------------------------
1 file changed, 254 insertions(+), 500 deletions(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 6cf5e2b..124c8b1 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -13,40 +13,230 @@
// License for the specific language governing permissions and limitations under
// the License.
-build_and_test = '''
-mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}
-cd ${platform}
-rm -rf build
-mkdir build
-cd build
-tar -xf ${WORKSPACE}/apache-couchdb-*.tar.gz
-cd apache-couchdb-*
-./configure --spidermonkey-version ${sm_ver}
-make check || (make build-report && false)
-'''
-
-make_packages = '''
-cd ${platform}
-git clone https://github.com/apache/couchdb-pkg
-rm -rf couchdb
-mkdir couchdb
-cp ${WORKSPACE}/apache-couchdb-*.tar.gz couchdb
-tar -xf ${WORKSPACE}/apache-couchdb-*.tar.gz -C couchdb
-cd couchdb-pkg
-make
-'''
-
-cleanup_and_save = '''
-rm -rf ${WORKSPACE}/pkgs/${platform}
-mkdir -p ${WORKSPACE}/pkgs/${platform}
-mv ${WORKSPACE}/${platform}/rpmbuild/RPMS/$(arch)/*rpm ${WORKSPACE}/pkgs/${platform} || true
-mv ${WORKSPACE}/${platform}/couchdb/*.deb ${WORKSPACE}/pkgs/${platform} || true
-'''
-
-update_qemu = '''
-docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
-'''
+// Erlang version embedded in binary packages
+ERLANG_VERSION = '24.2'
+
+// Erlang version used for rebar in release process. CouchDB will not build from
+// the release tarball on Erlang versions older than this
+MINIMUM_ERLANG_VERSION = '21'
+
+// We create parallel build / test / package stages for each OS using the metadata
+// in this map. Adding a new OS should ideally only involve adding a new entry here.
+meta = [
+ 'centos7': [
+ name: 'CentOS 7',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-centos:7-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'centos8': [
+ name: 'CentOS 8',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-centos:8-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'bionic': [
+ name: 'Ubuntu 18.04',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-ubuntu:bionic-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'focal': [
+ name: 'Ubuntu 20.04',
+ spidermonkey_vsn: '68',
+ image: "apache/couchdbci-ubuntu:focal-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'stretch': [
+ name: 'Debian 9',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-debian:stretch-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'buster': [
+ name: 'Debian 10',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-debian:buster-erlang-${ERLANG_VERSION}"
+ ],
+
+ // - Removed 2020.09.15 - VMs are offline
+
+ // 'buster-arm64': [
+ // name: 'Debian 10 ARM'
+ // spidermonkey_vsn: '60',
+ // image: "apache/couchdbci-debian:arm64v8-buster-erlang-${ERLANG_VERSION}",
+ // node_label: 'arm64v8'
+ // ],
+
+ // 'buster-ppc64': [
+ // name: 'Debian 10 POWER'
+ // spidermonkey_vsn: '60',
+ // image: "apache/couchdbci-debian:pp64le-buster-erlang-${ERLANG_VERSION}",
+ // node_label: 'ppc64le'
+ // ],
+
+ 'bullseye': [
+ name: 'Debian 11',
+ spidermonkey_vsn: '78',
+ image: "apache/couchdbci-debian:bullseye-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'freebsd': [
+ name: 'FreeBSD',
+ spidermonkey_vsn: '1.8.5',
+ gnu_make: 'gmake'
+ ],
+
+ 'macos': [
+ name: 'macOS',
+ spidermonkey_vsn: '60',
+ gnu_make: 'make'
+ ]
+]
+
+// Credit to https://stackoverflow.com/a/69222555 for this technique.
+// We can use the scripted pipeline syntax to dynamically generate stages,
+// and inject them into a map that we pass to the `parallel` step in a script.
+// While the scripting approach is very flexible, it's not able to use some
+// functionality specific to Declarative Pipelines, like the `agent` and `post`
+// directives, so you'll see alternatives like try-catch-finally used for flow
+// control and the nested `node` and `docker` blocks in the container stage to
+// configure the worker environment.
+
+// Returns a build stage suitable for non-containerized environments (currently
+// macOS and FreeBSD). Coincidentally we do not currently support automated
+// package generation on these platforms. This method in invoked when we create
+// `parallelStagesMap` below.
+def generateNativeStage(platform) {
+ return {
+ stage(platform) {
+ node(platform) {
+ timeout(time: 90, unit: "MINUTES") {
+ try {
+ // deleteDir is OK here because we're not inside of a Docker container!
+ deleteDir()
+ unstash 'tarball'
+ withEnv([
+ 'HOME='+pwd(),
+ 'PATH+USRLOCAL=/usr/local/bin',
+ 'MAKE='+meta[platform].gnu_make
+ ]) {
+ sh( script: "mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}/build", label: 'Create build directories' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/build --strip-components=1", label: 'Unpack release' )
+ dir( "${platform}/build" ) {
+ sh "./configure --skip-deps --spidermonkey-version ${meta[platform].spidermonkey_vsn}"
+ sh '$MAKE'
+ sh '$MAKE eunit'
+ sh '$MAKE elixir-suite'
+ sh '$MAKE exunit'
+ sh '$MAKE mango-test'
+ }
+ }
+ }
+ catch (err) {
+ echo "Build failed: {$err}"
+ currentBuild.result = 'FAILURE'
+ sh 'ls -l ${WORKSPACE}'
+ withEnv([
+ 'HOME='+pwd(),
+ 'PATH+USRLOCAL=/usr/local/bin',
+ 'MAKE='+meta[platform].gnu_make
+ ]) {
+ dir( "${platform}/build" ) {
+ sh 'ls -l'
+ sh '${MAKE} build-report'
+ }
+ }
+ }
+ finally {
+ junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
+ sh 'killall -9 beam.smp || true'
+ sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR}'
+ }
+ }
+ }
+ }
+ }
+}
+
+// Returns a build stage suitable for container-based deployments. This method
+// is invoked when we create the `parallelStagesMap` in the pipeline below.
+def generateContainerStage(platform) {
+ return {
+ // Important: the stage name here must match the parallelStagesMap key for the
+ // Jenkins UI to render the pipeline stages correctly. Don't ask why. -APK
+ stage(platform) {
+ node(meta[platform].get('node_label', 'docker')) {
+ docker.withRegistry('https://docker.io/', 'dockerhub_creds') {
+ docker.image(meta[platform].image).inside("${DOCKER_ARGS}") {
+ timeout(time: 90, unit: "MINUTES") {
+ stage("${meta[platform].name} - build & test") {
+ try {
+ sh( script: "rm -rf ${platform} apache-couchdb-*", label: 'Clean workspace' )
+ unstash 'tarball'
+ sh( script: "mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}/build", label: 'Create build directories' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/build --strip-components=1", label: 'Unpack release' )
+ dir( "${platform}/build" ) {
+ sh "./configure --skip-deps --spidermonkey-version ${meta[platform].spidermonkey_vsn}"
+ sh 'make'
+ sh 'make eunit'
+ sh 'make elixir-suite'
+ sh 'make exunit'
+ sh 'make mango-test'
+ }
+ }
+ catch (err) {
+ echo "Build failed: {$err}"
+ currentBuild.result = 'FAILURE'
+ sh 'ls -l ${WORKSPACE}'
+ dir( "${platform}/build" ) {
+ sh 'ls -l'
+ sh 'make build-report'
+ }
+ }
+ finally {
+ junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
+ sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR}'
+ }
+ }
+
+ stage("${meta[platform].name} - package") {
+ try {
+ unstash 'tarball'
+ sh( script: "mkdir -p ${platform}/couchdb", label: 'Create build directory' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/couchdb", label: 'Unpack release' )
+ sh( script: "cd ${platform} && git clone https://github.com/apache/couchdb-pkg", label: 'Clone packaging helper repo' )
+ dir( "${platform}/couchdb-pkg" ) {
+ sh( script: 'make', label: 'Build packages' )
+ }
+ sh( label: 'Stage package artifacts for archival', script: """
+ rm -rf pkgs/${platform}
+ mkdir -p pkgs/${platform}
+ mv ${platform}/rpmbuild/RPMS/\$(arch)/*rpm pkgs/${platform} || true
+ mv ${platform}/couchdb/*.deb pkgs/${platform} || true
+ """ )
+ archiveArtifacts artifacts: 'pkgs/**', fingerprint: true, onlyIfSuccessful: true
+ }
+ catch (err) {
+ echo "Packaging failed: {$err}"
+ currentBuild.result = 'FAILURE'
+ sh 'ls -l ${WORKSPACE}'
+ }
+ finally {
+ sh 'rm -rf ${WORKSPACE}/*'
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+}
+// Finally we have the actual Pipeline. It's mostly a Declarative Pipeline,
+// except for the 'Test and Package' stage where we use the `script` step as an
+// "escape hatch" to dynamically generate a set of parallel stages to execute.
pipeline {
// no top-level agent; agents must be declared for each stage
@@ -68,8 +258,6 @@ pipeline {
options {
buildDiscarder(logRotator(numToKeepStr: '10', artifactNumToKeepStr: '10'))
- // This fails the build immediately if any parallel step fails
- parallelsAlwaysFailFast()
preserveStashes(buildCount: 10)
timeout(time: 3, unit: 'HOURS')
timestamps()
@@ -80,30 +268,34 @@ pipeline {
agent {
docker {
label 'docker'
- image 'apache/couchdbci-debian:buster-erlang-21.3.8.17-1'
+ image "apache/couchdbci-debian:erlang-${MINIMUM_ERLANG_VERSION}"
args "${DOCKER_ARGS}"
registryUrl 'https://docker.io/'
registryCredentialsId 'dockerhub_creds'
}
}
+ environment {
+ // TODO find a way to avoid setting this explicitly
+ spidermonkey = '78'
+ }
options {
timeout(time: 15, unit: "MINUTES")
}
steps {
- sh '''
- set
- rm -rf apache-couchdb-*
- ./configure
- make erlfmt-check
- make dist
- chmod -R a+w * .
- '''
+ sh (script: 'rm -rf apache-couchdb-*', label: 'Clean workspace of any previous release artifacts' )
+ sh "./configure --spidermonkey-version ${spidermonkey}"
+ sh 'make erlfmt-check'
+ sh 'make elixir-check-formatted'
+ sh 'make dist'
}
post {
success {
stash includes: 'apache-couchdb-*.tar.gz', name: 'tarball'
archiveArtifacts artifacts: 'apache-couchdb-*.tar.gz', fingerprint: true
}
+ failure {
+ sh 'ls -l ${WORKSPACE}'
+ }
cleanup {
// UGH see https://issues.jenkins-ci.org/browse/JENKINS-41894
sh 'rm -rf ${WORKSPACE}/*'
@@ -111,457 +303,22 @@ pipeline {
}
} // stage Build Release Tarball
- // TODO Rework once Improved Docker Pipeline Engine is released
- // https://issues.jenkins-ci.org/browse/JENKINS-47962
- // https://issues.jenkins-ci.org/browse/JENKINS-48050
-
stage('Test and Package') {
-
- options {
- skipDefaultCheckout()
- timeout(time: 90, unit: "MINUTES")
- }
-
- parallel {
-
- stage('FreeBSD') {
- agent {
- label 'freebsd'
- }
- steps {
- // deleteDir is OK here because we're not inside of a Docker container!
- deleteDir()
- unstash 'tarball'
- withEnv(['HOME='+pwd()]) {
- sh '''
- mkdir -p $COUCHDB_IO_LOG_DIR
-
- # Build CouchDB from tarball & test
- mkdir build
- cd build
- tar -xf $WORKSPACE/apache-couchdb-*.tar.gz
- cd apache-couchdb-*
- ./configure
- gmake check || (build-aux/logfile-uploader.py && false)
-
- # No package build for FreeBSD at this time
- '''
- } // withEnv
- } // steps
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- cleanup {
- sh 'killall -9 beam.smp || true'
- sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR} || true'
- }
- } // post
- } // stage FreeBSD
-
- stage('macOS') {
- agent {
- label 'macos'
- }
- steps {
- // deleteDir is OK here because we're not inside of a Docker container!
- deleteDir()
- unstash 'tarball'
- withEnv(['HOME='+pwd()]) {
- sh '''
- PATH=/usr/local/bin:$PATH
- export PATH
- mkdir -p $COUCHDB_IO_LOG_DIR
-
- # Build CouchDB from tarball & test
- mkdir build
- cd build
- tar -xzf $WORKSPACE/apache-couchdb-*.tar.gz
- cd apache-couchdb-*
- ./configure --spidermonkey-version 60
- make check || (build-aux/logfile-uploader.py && false)
-
- # No package build for macOS at this time
- '''
- } // withEnv
- } // steps
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- cleanup {
- sh 'killall -9 beam.smp || true'
- sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR} || true'
- }
- } // post
- } // stage macOS
-
- stage('CentOS 7') {
- agent {
- docker {
- image 'apache/couchdbci-centos:7-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'centos7'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- unstash 'tarball'
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('CentOS 8') {
- agent {
- docker {
- image 'apache/couchdbci-centos:8-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'centos8'
- sm_ver = '60'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- unstash 'tarball'
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Ubuntu Bionic') {
- agent {
- docker {
- image 'apache/couchdbci-ubuntu:bionic-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'bionic'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Ubuntu Focal') {
- agent {
- docker {
- image 'apache/couchdbci-ubuntu:focal-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'focal'
- sm_ver = '68'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Debian Stretch') {
- agent {
- docker {
- image 'apache/couchdbci-debian:stretch-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'stretch'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Debian Buster amd64') {
- agent {
- docker {
- image 'apache/couchdbci-debian:buster-erlang-21.3.8.17-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'buster'
- sm_ver = '60'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
+ steps {
+ script {
+ // Including failFast: true in map fails the build immediately if any parallel step fails
+ parallelStagesMap = meta.collectEntries( [failFast: true] ) { key, values ->
+ if (values.image) {
+ ["${key}": generateContainerStage(key)]
}
- } // post
- } // stage
-
- stage('Debian Buster arm64v8') {
- when { expression { return false } }
- agent {
- docker {
- image 'apache/couchdbci-debian:arm64v8-buster-erlang-21.3.8.17-1'
- label 'arm64v8'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
+ else {
+ ["${key}": generateNativeStage(key)]
}
}
- environment {
- platform = 'buster'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
-/*
- - Removed 2020.09.15 - VMs are offline
-*/
-
-// stage('Debian Buster ppc64le') {
-// agent {
-// docker {
-// image 'apache/couchdbci-debian:ppc64le-buster-erlang-21.3.8.17-1'
-// label 'ppc64le'
-// args "${DOCKER_ARGS}"
-// registryUrl 'https://docker.io/'
-// registryCredentialsId 'dockerhub_creds'
-// }
-// }
-// environment {
-// platform = 'buster'
-// sm_ver = '60'
-// }
-// stages {
-// stage('Build from tarball & test') {
-// steps {
-// unstash 'tarball'
-// sh( script: build_and_test )
-// }
-// post {
-// always {
-// junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
-// }
-// }
-// }
-// stage('Build CouchDB packages') {
-// steps {
-// sh( script: make_packages )
-// sh( script: cleanup_and_save )
-// }
-// post {
-// success {
-// archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
-// }
-// }
-// }
-// } // stages
-// post {
-// cleanup {
-// sh 'rm -rf ${WORKSPACE}/*'
-// }
-// } // post
-// } // stage
+ parallel parallelStagesMap
+ }
+ }
+ }
/*
* Example of how to do a qemu-based run, please leave here
@@ -586,17 +343,17 @@ pipeline {
stages {
stage('Install latest qemu binaries') {
steps {
- sh( script: update_qemu )
+ sh( script: 'docker run --rm --privileged multiarch/qemu-user-static --reset -p yes' )
}
}
stage('Pull latest docker image') {
steps {
- sh "docker pull apache/couchdbci-debian:arm64v8-buster-erlang-21.3.8.17-1"
+ sh "docker pull apache/couchdbci-debian:arm64v8-buster-erlang-${ERLANG_VERSION}"
}
}
stage('Build from tarball & test & packages') {
steps {
- withDockerContainer(image: "apache/couchdbci-debian:arm64v8-buster-erlang-21.3.8.17-1", args: "${DOCKER_ARGS}") {
+ withDockerContainer(image: "apache/couchdbci-debian:arm64v8-buster-erlang-${ERLANG_VERSION}", args: "${DOCKER_ARGS}") {
unstash 'tarball'
withEnv(['MIX_HOME='+pwd(), 'HEX_HOME='+pwd()]) {
sh( script: build_and_test )
@@ -625,9 +382,6 @@ pipeline {
} // stage
*/
- } // parallel
- } // stage "Test and Package"
-
stage('Publish') {
when {
@@ -636,7 +390,7 @@ pipeline {
agent {
docker {
- image 'apache/couchdbci-debian:buster-erlang-21.3.8.17-1'
+ image "apache/couchdbci-debian:erlang-${ERLANG_VERSION}"
label 'docker'
args "${DOCKER_ARGS}"
registryUrl 'https://docker.io/'
@@ -678,8 +432,8 @@ pipeline {
sh '''
cp js/centos-7/*rpm pkgs/centos7
cp js/centos-8/*rpm pkgs/centos8
- cd pkgs/centos7 && createrepo --database .
- cd ../centos8 && createrepo --database .
+ cd pkgs/centos7 && createrepo_c --database .
+ cd ../centos8 && createrepo_c --database .
'''
echo 'Building tree to upload...'
[couchdb] 01/02: Tweak Elixir formatting
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 040f03bc69272a0cc5f5322684fd4e70b68a067e
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Sat Jan 15 21:02:05 2022 +0000
Tweak Elixir formatting
It seems different versions of mix cannot agree on how these lines
ought to be formatted, so let's try to help out.
---
src/chttpd/test/exunit/pagination_test.exs | 61 +++++++++++++-----------------
1 file changed, 26 insertions(+), 35 deletions(-)
diff --git a/src/chttpd/test/exunit/pagination_test.exs b/src/chttpd/test/exunit/pagination_test.exs
index f071036..cd541dc 100644
--- a/src/chttpd/test/exunit/pagination_test.exs
+++ b/src/chttpd/test/exunit/pagination_test.exs
@@ -580,9 +580,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : _all_docs?page_size=#{n}&descending=#{
- descending
- }" do
+ describe "Pagination API (10 docs) : _all_docs" <>
+ "?page_size=#{n}&descending=#{descending}" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -613,9 +612,8 @@ defmodule Couch.Test.Pagination do
end
end
- describe "Pagination API (10 docs) : _all_docs?page_size=#{n}&descending=#{
- descending
- } : range" do
+ describe "Pagination API (10 docs) : _all_docs" <>
+ "?page_size=#{n}&descending=#{descending} : range" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -706,9 +704,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : _all_docs?page_size=#{n}&descending=#{
- descending
- } : pages" do
+ describe "Pagination API (10 docs) : _all_docs" <>
+ "?page_size=#{n}&descending=#{descending} : pages" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -752,7 +749,8 @@ defmodule Couch.Test.Pagination do
end
for n <- 10..11 do
- describe "Pagination API (10 docs) : _all_docs?page_size=#{n}" do
+ describe "Pagination API (10 docs) : _all_docs" <>
+ "?page_size=#{n}" do
@describetag n_docs: 10
@describetag descending: false
@describetag page_size: n
@@ -772,9 +770,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : _all_docs/queries?page_size=#{n}&descending=#{
- descending
- } : pages" do
+ describe "Pagination API (10 docs) : _all_docs/queries" <>
+ "?page_size=#{n}&descending=#{descending} : pages" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -908,9 +905,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view?page_size=#{n}&descending=#{
- descending
- }" do
+ describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view" <>
+ "?page_size=#{n}&descending=#{descending}" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -1015,7 +1011,8 @@ defmodule Couch.Test.Pagination do
end
for n <- 10..11 do
- describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view?page_size=#{n}" do
+ describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view" <>
+ "?page_size=#{n}" do
@describetag n_docs: 10
@describetag descending: false
@describetag page_size: n
@@ -1062,9 +1059,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view/queries?page_size=#{
- n
- }&descending=#{descending} : pages" do
+ describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view/queries" <>
+ "?page_size=#{n}&descending=#{descending} : pages" do
@describetag n_docs: 10
@describetag descending: descending
@describetag page_size: n
@@ -1224,14 +1220,12 @@ defmodule Couch.Test.Pagination do
expected_ids_order = :ascending
assert expected_key_order == ordering?(results, "key"),
- "expecting keys in #{expected_key_order} order, got: #{
- inspect(field(results, "key"))
- }"
+ "expecting keys in #{expected_key_order} order, " <>
+ "got: #{inspect(field(results, "key"))}"
assert expected_ids_order == ordering?(results, "id"),
- "expecting ids in #{expected_ids_order} order, got: #{
- inspect(field(results, "id"))
- }"
+ "expecting ids in #{expected_ids_order} order, " <>
+ "got: #{inspect(field(results, "id"))}"
results = List.flatten(limit_query)
[_descendiing_query, query] = ctx.queries[:queries]
@@ -1253,14 +1247,12 @@ defmodule Couch.Test.Pagination do
end
assert expected_key_order == ordering?(results, "key"),
- ~s(expecting keys in #{expected_key_order} order, got: #{
- inspect(field(results, "key"))
- })
+ "expecting keys in #{expected_key_order} order, " <>
+ "got: #{inspect(field(results, "key"))}"
assert expected_ids_order == ordering?(results, "id"),
- ~s(expecting keys in #{expected_ids_order} order, got: #{
- inspect(field(results, "id"))
- })
+ "expecting keys in #{expected_ids_order} order, " <>
+ "got: #{inspect(field(results, "id"))}"
_keys = Enum.map(results, &Map.get(&1, "key"))
end
@@ -1270,9 +1262,8 @@ defmodule Couch.Test.Pagination do
for descending <- [false, true] do
for n <- [4, 9] do
- describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view/queries?page_size=#{
- n
- }&descending=#{descending} : pages with same key" do
+ describe "Pagination API (10 docs) : /{db}/_design/{ddoc}/_view/queries" <>
+ "?page_size=#{n}&descending=#{descending} : pages with same key" do
@describetag descending: descending
@describetag n_docs: 10
@describetag page_size: n