You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by ko...@apache.org on 2022/01/21 02:47:54 UTC
[couchdb] branch jenkins-dynamic-matrix-3x updated (cbee041 -> df818b6)
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a change to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git.
discard cbee041 Try to figure out why tree is dirty
discard 8ea19b7 Fix SM60 prohibition on ARM
discard 4d6c511 Bump ARM and POWER to Debian 11 / Erlang 24
discard f1bb1a7 More fixes
discard e899a8c Run same suites as 'check' target
discard e276402 Forgot Erlang 20 was still SM_VSN=60
discard 16e4145 Refactor Jenkins to dynamically generate stages
add fedf37d Non-zero instance_start_time
add 3abcc42 Merge pull request #3901 from apache/nonzero_instance_start_time
add 63a9d45 Enable exunit in check target
add ab700ba Do not start test name with "test "
add 55d8278 Fix doc order in fabric_doc_update tests
new f0994aa Refactor Jenkins to dynamically generate stages
new a99c7ba Forgot Erlang 20 was still SM_VSN=60
new 9fd6b45 Run same suites as 'check' target
new 1c57cdd More fixes
new eabc373 Bump ARM and POWER to Debian 11 / Erlang 24
new 78a152f Fix SM60 prohibition on ARM
new 7679624 Try to figure out why tree is dirty
new 3783cf7 Restore ExUnit test
new df818b6 Try to find the dirt again
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cbee041)
\
N -- N -- N refs/heads/jenkins-dynamic-matrix-3x (df818b6)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Makefile | 2 ++
build-aux/Jenkinsfile.full | 4 +++-
src/chttpd/src/chttpd_db.erl | 3 ++-
.../src/couch_replicator_scheduler_job.erl | 9 +++------
src/fabric/src/fabric_db_info.erl | 5 +++--
src/fabric/src/fabric_doc_update.erl | 10 +++++-----
src/mem3/src/mem3.erl | 15 +++++++++++++++
test/elixir/test/attachments_test.exs | 2 +-
test/elixir/test/design_docs_test.exs | 2 +-
9 files changed, 35 insertions(+), 17 deletions(-)
[couchdb] 09/09: Try to find the dirt again
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit df818b6367ed1129bd42313f3209d8ec3d12fefc
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 21:46:42 2022 -0500
Try to find the dirt again
---
build-aux/Jenkinsfile.full | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 8c2ea95..d1e0fa6 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -278,10 +278,10 @@ pipeline {
steps {
timeout(time: 15, unit: "MINUTES") {
sh (script: 'rm -rf apache-couchdb-*', label: 'Clean workspace of any previous release artifacts' )
- sh 'git status'
sh "./configure --spidermonkey-version ${spidermonkey}"
sh 'make erlfmt-check'
sh 'make elixir-check-formatted'
+ sh 'git status'
sh 'make dist'
}
}
[couchdb] 08/09: Restore ExUnit test
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 3783cf79dd8e0b24e449ac6643625aded9b20042
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 21:45:16 2022 -0500
Restore ExUnit test
---
build-aux/Jenkinsfile.full | 2 ++
1 file changed, 2 insertions(+)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index d1c0f19..8c2ea95 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -126,6 +126,7 @@ def generateNativeStage(platform) {
sh '$MAKE'
sh '$MAKE eunit'
sh '$MAKE elixir-suite'
+ sh '$MAKE exunit'
sh '$MAKE mango-test'
sh '$MAKE weatherreport-test'
}
@@ -178,6 +179,7 @@ def generateContainerStage(platform) {
sh 'make'
sh 'make eunit'
sh 'make elixir-suite'
+ sh 'make exunit'
sh 'make mango-test'
sh 'make weatherreport-test'
}
[couchdb] 02/09: Forgot Erlang 20 was still SM_VSN=60
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit a99c7ba854ddafea9a8d7a4493152bd10e1af8ce
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 09:21:53 2022 -0500
Forgot Erlang 20 was still SM_VSN=60
---
build-aux/Jenkinsfile.full | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 4b0e232..2315e03 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -271,7 +271,7 @@ pipeline {
}
environment {
// TODO find a way to avoid setting this explicitly
- spidermonkey = '78'
+ spidermonkey = '60'
}
steps {
timeout(time: 15, unit: "MINUTES") {
[couchdb] 05/09: Bump ARM and POWER to Debian 11 / Erlang 24
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit eabc37377ffb18a945f55ef34ea305dd30e38fc6
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 19:40:23 2022 -0500
Bump ARM and POWER to Debian 11 / Erlang 24
Use the buildx-based multi-arch container image.
---
build-aux/Jenkinsfile.full | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 88de754..884951c 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -59,17 +59,17 @@ meta = [
image: "apache/couchdbci-debian:buster-erlang-${ERLANG_VERSION}"
],
- 'buster-arm64': [
- name: 'Debian 10 ARM',
- spidermonkey_vsn: '1.8.5',
- image: "apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1",
+ 'bullseye-arm64': [
+ name: 'Debian 11 ARM',
+ spidermonkey_vsn: '78',
+ image: "apache/couchdbci-debian:bullseye-erlang-${ERLANG_VERSION}",
node_label: 'arm64v8'
],
- 'buster-ppc64': [
- name: 'Debian 10 POWER',
- spidermonkey_vsn: '60',
- image: "apache/couchdbci-debian:ppc64le-buster-erlang-20.3.8.26-1",
+ 'bullseye-ppc64': [
+ name: 'Debian 11 POWER',
+ spidermonkey_vsn: '78',
+ image: "apache/couchdbci-debian:bullseye-erlang-${ERLANG_VERSION}",
node_label: 'ppc64le'
],
[couchdb] 04/09: More fixes
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 1c57cdd9429d92622acc49e0324961f747b77dbd
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 15:03:41 2022 -0500
More fixes
---
build-aux/Jenkinsfile.full | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index c9dac11..88de754 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -61,7 +61,7 @@ meta = [
'buster-arm64': [
name: 'Debian 10 ARM',
- spidermonkey_vsn: '60',
+ spidermonkey_vsn: '1.8.5',
image: "apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1",
node_label: 'arm64v8'
],
@@ -69,7 +69,7 @@ meta = [
'buster-ppc64': [
name: 'Debian 10 POWER',
spidermonkey_vsn: '60',
- image: "apache/couchdbci-debian:pp64le-buster-erlang-20.3.8.26-1",
+ image: "apache/couchdbci-debian:ppc64le-buster-erlang-20.3.8.26-1",
node_label: 'ppc64le'
],
[couchdb] 01/09: Refactor Jenkins to dynamically generate stages
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit f0994aa42ff797cab6061599924d6f28751659e7
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Wed Jan 19 22:37:44 2022 -0500
Refactor Jenkins to dynamically generate stages
This is one of those situations where you go in to make a small change,
see an opportunity for some refactoring, and get sucked into a rabbit
hole that leaves you wondering if you have any idea how computers
actually work. My initial goal was simply to update the Erlang version
used in our binary packages to a modern supported release. Along the
way I decided I wanted to figure out how to eliminate all the copypasta
we generate for making any change to this file, and after a few days of
hacking here we are. This rewrite has the following features:
* Updates to use Debian 11 (current stable) as the base image for
building releases and packaging repos.
* Defaults to Erlang 24.2 as the embedded Erlang version in packages.
* Dynamically generates the parallel build stages used to test and
package CouchDB on various OSes. This is accomplished through a bit
of scripted pipeline code that relies on two new methods defined at
the beginning of the Jenkinsfile, one for "native" builds on macOS
and FreeBSD and one for container-based builds. See comments in the
Jenkinsfile for additional details.
* Expands commands like `make check` into a series of steps to improve
visibility. The Jenkins UI will now show the time spent in each step
of the build process, and if a step (e.g. `make eunit`) fails it will
only expand the logs for that step by default instead of showing the
logs for the entire build stage. The downside is that if we do make
changes to the series of targets underneath `check` we need to
remember to update the Jenkinsfile as well.
* Starts per-stage timer _after_ agent is acquired. Previously builds could
fail with a 15m timeout when all they did was sit in the build queue.
This is a cherry-pick of 9b6454b from main, modified to drop the
MINIMUM_ERLANG_VERSION to 20 and adding ARM and POWER back into the
matrix. Still need to build newer container images for those
architectures, though, so for now they're still Buster / Erlang 20.
---
build-aux/Jenkinsfile.full | 792 ++++++++++----------------------
build-aux/{README.Jenkins => README.md} | 70 ++-
2 files changed, 276 insertions(+), 586 deletions(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 37f49e2..4b0e232 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -13,40 +13,225 @@
// License for the specific language governing permissions and limitations under
// the License.
-build_and_test = '''
-mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}
-cd ${platform}
-rm -rf build
-mkdir build
-cd build
-tar -xf ${WORKSPACE}/apache-couchdb-*.tar.gz
-cd apache-couchdb-*
-./configure --spidermonkey-version ${sm_ver}
-make check || (make build-report && false)
-'''
-
-make_packages = '''
-cd ${platform}
-git clone https://github.com/apache/couchdb-pkg
-rm -rf couchdb
-mkdir couchdb
-cp ${WORKSPACE}/apache-couchdb-*.tar.gz couchdb
-tar -xf ${WORKSPACE}/apache-couchdb-*.tar.gz -C couchdb
-cd couchdb-pkg
-make
-'''
-
-cleanup_and_save = '''
-rm -rf ${WORKSPACE}/pkgs/${platform}
-mkdir -p ${WORKSPACE}/pkgs/${platform}
-mv ${WORKSPACE}/${platform}/rpmbuild/RPMS/$(arch)/*rpm ${WORKSPACE}/pkgs/${platform} || true
-mv ${WORKSPACE}/${platform}/couchdb/*.deb ${WORKSPACE}/pkgs/${platform} || true
-'''
-
-update_qemu = '''
-docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
-'''
+// Erlang version embedded in binary packages
+ERLANG_VERSION = '24.2'
+
+// Erlang version used for rebar in release process. CouchDB will not build from
+// the release tarball on Erlang versions older than this
+MINIMUM_ERLANG_VERSION = '20'
+
+// We create parallel build / test / package stages for each OS using the metadata
+// in this map. Adding a new OS should ideally only involve adding a new entry here.
+meta = [
+ 'centos7': [
+ name: 'CentOS 7',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-centos:7-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'centos8': [
+ name: 'CentOS 8',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-centos:8-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'bionic': [
+ name: 'Ubuntu 18.04',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-ubuntu:bionic-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'focal': [
+ name: 'Ubuntu 20.04',
+ spidermonkey_vsn: '68',
+ image: "apache/couchdbci-ubuntu:focal-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'stretch': [
+ name: 'Debian 9',
+ spidermonkey_vsn: '1.8.5',
+ image: "apache/couchdbci-debian:stretch-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'buster': [
+ name: 'Debian 10',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-debian:buster-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'buster-arm64': [
+ name: 'Debian 10 ARM',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1",
+ node_label: 'arm64v8'
+ ],
+
+ 'buster-ppc64': [
+ name: 'Debian 10 POWER',
+ spidermonkey_vsn: '60',
+ image: "apache/couchdbci-debian:pp64le-buster-erlang-20.3.8.26-1",
+ node_label: 'ppc64le'
+ ],
+
+ 'bullseye': [
+ name: 'Debian 11',
+ spidermonkey_vsn: '78',
+ image: "apache/couchdbci-debian:bullseye-erlang-${ERLANG_VERSION}"
+ ],
+
+ 'freebsd': [
+ name: 'FreeBSD',
+ spidermonkey_vsn: '1.8.5',
+ gnu_make: 'gmake'
+ ],
+
+ 'macos': [
+ name: 'macOS',
+ spidermonkey_vsn: '60',
+ gnu_make: 'make'
+ ]
+]
+
+// Credit to https://stackoverflow.com/a/69222555 for this technique.
+// We can use the scripted pipeline syntax to dynamically generate stages,
+// and inject them into a map that we pass to the `parallel` step in a script.
+// While the scripting approach is very flexible, it's not able to use some
+// functionality specific to Declarative Pipelines, like the `agent` and `post`
+// directives, so you'll see alternatives like try-catch-finally used for flow
+// control and the nested `node` and `docker` blocks in the container stage to
+// configure the worker environment.
+
+// Returns a build stage suitable for non-containerized environments (currently
+// macOS and FreeBSD). Coincidentally we do not currently support automated
+// package generation on these platforms. This method in invoked when we create
+// `parallelStagesMap` below.
+def generateNativeStage(platform) {
+ return {
+ stage(platform) {
+ node(platform) {
+ timeout(time: 90, unit: "MINUTES") {
+ try {
+ // deleteDir is OK here because we're not inside of a Docker container!
+ deleteDir()
+ unstash 'tarball'
+ withEnv([
+ 'HOME='+pwd(),
+ 'PATH+USRLOCAL=/usr/local/bin',
+ 'MAKE='+meta[platform].gnu_make
+ ]) {
+ sh( script: "mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}/build", label: 'Create build directories' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/build --strip-components=1", label: 'Unpack release' )
+ dir( "${platform}/build" ) {
+ sh "./configure --skip-deps --spidermonkey-version ${meta[platform].spidermonkey_vsn}"
+ sh '$MAKE'
+ sh '$MAKE eunit'
+ sh '$MAKE elixir-suite'
+ sh '$MAKE exunit'
+ sh '$MAKE mango-test'
+ }
+ }
+ }
+ catch (err) {
+ sh 'ls -l ${WORKSPACE}'
+ withEnv([
+ 'HOME='+pwd(),
+ 'PATH+USRLOCAL=/usr/local/bin',
+ 'MAKE='+meta[platform].gnu_make
+ ]) {
+ dir( "${platform}/build" ) {
+ sh 'ls -l'
+ sh '${MAKE} build-report'
+ }
+ }
+ error("Build step failed with error: ${err.getMessage()}")
+ }
+ finally {
+ junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
+ sh 'killall -9 beam.smp || true'
+ sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR}'
+ }
+ }
+ }
+ }
+ }
+}
+
+// Returns a build stage suitable for container-based deployments. This method
+// is invoked when we create the `parallelStagesMap` in the pipeline below.
+def generateContainerStage(platform) {
+ return {
+ // Important: the stage name here must match the parallelStagesMap key for the
+ // Jenkins UI to render the pipeline stages correctly. Don't ask why. -APK
+ stage(platform) {
+ node(meta[platform].get('node_label', 'docker')) {
+ docker.withRegistry('https://docker.io/', 'dockerhub_creds') {
+ docker.image(meta[platform].image).inside("${DOCKER_ARGS}") {
+ timeout(time: 90, unit: "MINUTES") {
+ stage("${meta[platform].name} - build & test") {
+ try {
+ sh( script: "rm -rf ${platform} apache-couchdb-*", label: 'Clean workspace' )
+ unstash 'tarball'
+ sh( script: "mkdir -p ${COUCHDB_IO_LOG_DIR} ${platform}/build", label: 'Create build directories' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/build --strip-components=1", label: 'Unpack release' )
+ dir( "${platform}/build" ) {
+ sh "./configure --skip-deps --spidermonkey-version ${meta[platform].spidermonkey_vsn}"
+ sh 'make'
+ sh 'make eunit'
+ sh 'make elixir-suite'
+ sh 'make exunit'
+ sh 'make mango-test'
+ }
+ }
+ catch (err) {
+ sh 'ls -l ${WORKSPACE}'
+ dir( "${platform}/build" ) {
+ sh 'ls -l'
+ sh 'make build-report'
+ }
+ error("Build step failed with error: ${err.getMessage()}")
+ }
+ finally {
+ junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
+ sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR}'
+ }
+ }
+
+ stage("${meta[platform].name} - package") {
+ try {
+ unstash 'tarball'
+ sh( script: "mkdir -p ${platform}/couchdb", label: 'Create build directory' )
+ sh( script: "tar -xf apache-couchdb-*.tar.gz -C ${platform}/couchdb", label: 'Unpack release' )
+ sh( script: "cd ${platform} && git clone https://github.com/apache/couchdb-pkg", label: 'Clone packaging helper repo' )
+ dir( "${platform}/couchdb-pkg" ) {
+ sh( script: 'make', label: 'Build packages' )
+ }
+ sh( label: 'Stage package artifacts for archival', script: """
+ rm -rf pkgs/${platform}
+ mkdir -p pkgs/${platform}
+ mv ${platform}/rpmbuild/RPMS/\$(arch)/*rpm pkgs/${platform} || true
+ mv ${platform}/couchdb/*.deb pkgs/${platform} || true
+ """ )
+ archiveArtifacts artifacts: 'pkgs/**', fingerprint: true, onlyIfSuccessful: true
+ }
+ catch (err) {
+ sh 'ls -l ${WORKSPACE}'
+ error("Build step failed with error: ${err.getMessage()}")
+ }
+ finally {
+ sh 'rm -rf ${WORKSPACE}/*'
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+}
+// Finally we have the actual Pipeline. It's mostly a Declarative Pipeline,
+// except for the 'Test and Package' stage where we use the `script` step as an
+// "escape hatch" to dynamically generate a set of parallel stages to execute.
pipeline {
// no top-level agent; agents must be declared for each stage
@@ -68,8 +253,6 @@ pipeline {
options {
buildDiscarder(logRotator(numToKeepStr: '10', artifactNumToKeepStr: '10'))
- // This fails the build immediately if any parallel step fails
- parallelsAlwaysFailFast()
preserveStashes(buildCount: 10)
timeout(time: 3, unit: 'HOURS')
timestamps()
@@ -80,29 +263,33 @@ pipeline {
agent {
docker {
label 'docker'
- image 'apache/couchdbci-debian:buster-erlang-20.3.8.26-1'
+ image "apache/couchdbci-debian:erlang-${MINIMUM_ERLANG_VERSION}"
args "${DOCKER_ARGS}"
registryUrl 'https://docker.io/'
registryCredentialsId 'dockerhub_creds'
}
}
- options {
- timeout(time: 15, unit: "MINUTES")
+ environment {
+ // TODO find a way to avoid setting this explicitly
+ spidermonkey = '78'
}
steps {
- sh '''
- set
- rm -rf apache-couchdb-*
- ./configure
- make dist
- chmod -R a+w * .
- '''
+ timeout(time: 15, unit: "MINUTES") {
+ sh (script: 'rm -rf apache-couchdb-*', label: 'Clean workspace of any previous release artifacts' )
+ sh "./configure --spidermonkey-version ${spidermonkey}"
+ sh 'make erlfmt-check'
+ sh 'make elixir-check-formatted'
+ sh 'make dist'
+ }
}
post {
success {
stash includes: 'apache-couchdb-*.tar.gz', name: 'tarball'
archiveArtifacts artifacts: 'apache-couchdb-*.tar.gz', fingerprint: true
}
+ failure {
+ sh 'ls -l ${WORKSPACE}'
+ }
cleanup {
// UGH see https://issues.jenkins-ci.org/browse/JENKINS-41894
sh 'rm -rf ${WORKSPACE}/*'
@@ -110,498 +297,22 @@ pipeline {
}
} // stage Build Release Tarball
- // TODO Rework once Improved Docker Pipeline Engine is released
- // https://issues.jenkins-ci.org/browse/JENKINS-47962
- // https://issues.jenkins-ci.org/browse/JENKINS-48050
-
stage('Test and Package') {
-
- options {
- skipDefaultCheckout()
- timeout(time: 90, unit: "MINUTES")
- }
-
- parallel {
-
- stage('FreeBSD') {
- agent {
- label 'freebsd'
- }
- steps {
- // deleteDir is OK here because we're not inside of a Docker container!
- deleteDir()
- unstash 'tarball'
- withEnv(['HOME='+pwd()]) {
- sh '''
- mkdir -p $COUCHDB_IO_LOG_DIR
-
- # Build CouchDB from tarball & test
- mkdir build
- cd build
- tar -xf $WORKSPACE/apache-couchdb-*.tar.gz
- cd apache-couchdb-*
- ./configure
- gmake check || (build-aux/logfile-uploader.py && false)
-
- # No package build for FreeBSD at this time
- '''
- } // withEnv
- } // steps
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- cleanup {
- sh 'killall -9 beam.smp || true'
- sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR} || true'
- }
- } // post
- } // stage FreeBSD
-
- stage('macOS') {
- agent {
- label 'macos'
- }
- steps {
- // deleteDir is OK here because we're not inside of a Docker container!
- deleteDir()
- unstash 'tarball'
- withEnv(['HOME='+pwd()]) {
- sh '''
- PATH=/usr/local/bin:$PATH
- export PATH
- mkdir -p $COUCHDB_IO_LOG_DIR
-
- # Build CouchDB from tarball & test
- mkdir build
- cd build
- tar -xzf $WORKSPACE/apache-couchdb-*.tar.gz
- cd apache-couchdb-*
- ./configure --spidermonkey-version 60
- make check || (build-aux/logfile-uploader.py && false)
-
- # No package build for macOS at this time
- '''
- } // withEnv
- } // steps
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- cleanup {
- sh 'killall -9 beam.smp || true'
- sh 'rm -rf ${WORKSPACE}/* ${COUCHDB_IO_LOG_DIR} || true'
- }
- } // post
- } // stage macOS
-
- stage('CentOS 7') {
- agent {
- docker {
- image 'apache/couchdbci-centos:7-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'centos7'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- unstash 'tarball'
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('CentOS 8') {
- agent {
- docker {
- image 'apache/couchdbci-centos:8-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'centos8'
- sm_ver = '60'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- unstash 'tarball'
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Ubuntu Bionic') {
- agent {
- docker {
- image 'apache/couchdbci-ubuntu:bionic-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'bionic'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Ubuntu Focal') {
- agent {
- docker {
- image 'apache/couchdbci-ubuntu:focal-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'focal'
- sm_ver = '68'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Debian Stretch') {
- agent {
- docker {
- image 'apache/couchdbci-debian:stretch-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'stretch'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Debian Buster amd64') {
- agent {
- docker {
- image 'apache/couchdbci-debian:buster-erlang-20.3.8.26-1'
- label 'docker'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'buster'
- sm_ver = '60'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
+ steps {
+ script {
+ // Including failFast: true in map fails the build immediately if any parallel step fails
+ parallelStagesMap = meta.collectEntries( [failFast: false] ) { key, values ->
+ if (values.image) {
+ ["${key}": generateContainerStage(key)]
}
- } // post
- } // stage
-
- stage('Debian Buster arm64v8') {
- agent {
- docker {
- image 'apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1'
- label 'arm64v8'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
+ else {
+ ["${key}": generateNativeStage(key)]
}
}
- environment {
- platform = 'buster'
- sm_ver = '1.8.5'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- stage('Debian Buster ppc64le') {
- agent {
- docker {
- image 'apache/couchdbci-debian:ppc64le-buster-erlang-20.3.8.26-1'
- label 'ppc64le'
- args "${DOCKER_ARGS}"
- registryUrl 'https://docker.io/'
- registryCredentialsId 'dockerhub_creds'
- }
- }
- environment {
- platform = 'buster'
- sm_ver = '60'
- }
- stages {
- stage('Build from tarball & test') {
- steps {
- unstash 'tarball'
- sh( script: build_and_test )
- }
- post {
- always {
- junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- }
- }
- }
- stage('Build CouchDB packages') {
- steps {
- sh( script: make_packages )
- sh( script: cleanup_and_save )
- }
- post {
- success {
- archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- }
- }
- }
- } // stages
- post {
- cleanup {
- sh 'rm -rf ${WORKSPACE}/*'
- }
- } // post
- } // stage
-
- // stage('Debian Buster s390x') {
- // agent {
- // docker {
- // image 'apache/couchdbci-debian:s390x-buster-erlang-20.3.8.26-1'
- // label 's390x'
- // args "${DOCKER_ARGS}"
- // registryUrl 'https://docker.io/'
- // registryCredentialsId 'dockerhub_creds'
- // }
- // }
- // environment {
- // platform = 'buster'
- // sm_ver = '60'
- // }
- // stages {
- // stage('Build from tarball & test') {
- // steps {
- // unstash 'tarball'
- // sh( script: build_and_test )
- // }
- // post {
- // always {
- // junit '**/.eunit/*.xml, **/_build/*/lib/couchdbtest/*.xml, **/src/mango/nosetests.xml, **/test/javascript/junit.xml'
- // }
- // }
- // }
- // stage('Build CouchDB packages') {
- // steps {
- // sh( script: make_packages )
- // sh( script: cleanup_and_save )
- // }
- // post {
- // success {
- // archiveArtifacts artifacts: 'pkgs/**', fingerprint: true
- // }
- // }
- // }
- // } // stages
- // post {
- // cleanup {
- // sh 'rm -rf ${WORKSPACE}/*'
- // }
- // } // post
- // } // stage
-
+ parallel parallelStagesMap
+ }
+ }
+ }
/*
* Example of how to do a qemu-based run, please leave here
@@ -626,17 +337,17 @@ pipeline {
stages {
stage('Install latest qemu binaries') {
steps {
- sh( script: update_qemu )
+ sh( script: 'docker run --rm --privileged multiarch/qemu-user-static --reset -p yes' )
}
}
stage('Pull latest docker image') {
steps {
- sh "docker pull apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1"
+ sh "docker pull apache/couchdbci-debian:arm64v8-buster-erlang-${ERLANG_VERSION}"
}
}
stage('Build from tarball & test & packages') {
steps {
- withDockerContainer(image: "apache/couchdbci-debian:arm64v8-buster-erlang-20.3.8.26-1", args: "${DOCKER_ARGS}") {
+ withDockerContainer(image: "apache/couchdbci-debian:arm64v8-buster-erlang-${ERLANG_VERSION}", args: "${DOCKER_ARGS}") {
unstash 'tarball'
withEnv(['MIX_HOME='+pwd(), 'HEX_HOME='+pwd()]) {
sh( script: build_and_test )
@@ -665,9 +376,6 @@ pipeline {
} // stage
*/
- } // parallel
- } // stage "Test and Package"
-
stage('Publish') {
when {
@@ -676,7 +384,7 @@ pipeline {
agent {
docker {
- image 'apache/couchdbci-debian:buster-erlang-20.3.8.26-1'
+ image "apache/couchdbci-debian:erlang-${ERLANG_VERSION}"
label 'docker'
args "${DOCKER_ARGS}"
registryUrl 'https://docker.io/'
@@ -718,8 +426,8 @@ pipeline {
sh '''
cp js/centos-7/*rpm pkgs/centos7
cp js/centos-8/*rpm pkgs/centos8
- cd pkgs/centos7 && createrepo --database .
- cd ../centos8 && createrepo --database .
+ cd pkgs/centos7 && createrepo_c --database .
+ cd ../centos8 && createrepo_c --database .
'''
echo 'Building tree to upload...'
diff --git a/build-aux/README.Jenkins b/build-aux/README.md
similarity index 70%
rename from build-aux/README.Jenkins
rename to build-aux/README.md
index 014fbaa..72cef94 100644
--- a/build-aux/README.Jenkins
+++ b/build-aux/README.md
@@ -1,40 +1,6 @@
-# Building with a dedicated Jenkins CI instance
+# CouchDB and Jenkins CI
-## History
-
-Around April 2019, once Travis CI performance became so abysmal that the
-development team couldn't tolerate it any longer. We decided to move off
-of it entirely and onto our own dedicated Jenkins instance. Performance
-was bad because:
-
-* The Travis CI VMs are severely underpowered, leading to many
- unexpected test failures
-* The ASF wait queue for Travis was exceedingly long, with jobs
- sometimes taking 30-45 minutes just to _launch_
-* Our test suites (all 3 of Erlang, JavaScript and Elixir) had
- sporadic failures, requiring many re-launches of builds before
- we could move forward
-
-We also had other reasons to want to move to a dedicated server:
-
-* Credentials on the main Jenkins instance could be reused by *anyone*
- in the ASF
-* Credentials on the Travis CI setup should not contain credentials
- for things such as building binaries, to ensure the best possible
- security
-
-Other options we explored:
-
-* ASF BuildBot. This required a lot more work and couldn't guarantee
- availability of sufficient workers.
-* Circle CI, GitHub CI, etc.: Lack of availability of alternate binary
- platforms we've been asked to support (arm64v8, ppc64le) or alternate
- OSes (OSX, FreeBSD, Windows), though there is some support for either
- or both.
-
-## Cloudbees Core
-
-End of November 2019, ASF and CloudBees reached an agreement to allow
+In 2019, ASF and CloudBees reached an agreement to allow
the Foundation to use CloudBees Core to have a farm of managed Jenkins
masters. This allows the ASF to give larger projects their own dedicated
Jenkins master, which can be custom configured for the project. They can
@@ -53,15 +19,15 @@ the project's exclusive use. Combined with the FreeBSD and OSX nodes the
project already had, this will provide the necessary compute resources
to meet the needs of the project for some time to come.
-# Jenkins configuration
+# Jenkins Configuration
All jobs on the new Jenkins master are contained in a CouchDB folder.
Credentials for the project are placed within this folder; this is a
unique capability of CloudBees Core at this time.
-## Pull Request job configuration
+## Pull Requests
-Blue Ocean link: https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/activity/
+[Blue Ocean link](https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/activity/)
To implement build-a-PR functionality in the same way that Travis
performs builds, Jenkins offers the Multibranch Pipeline job. Here's how
@@ -106,12 +72,15 @@ it's configured for CouchDB through the GUI:
[2]: https://issues.apache.org/jira/browse/INFRA-17449 explains why "Discover pull requests from forks/Trust" has been set to "Nobody."
-## `master` and release branch configuration
+## Full Platform Builds on `main` and release branches
-Our old Jenkins job (formerly `/Jenkinsfile`) is now
-`build-aux/Jenkinsfile.full`. This builds CouchDB on `master`, all of
-our release branches (`2.*`, `3.*`, etc.) as well as `jenkins-*` for
-testing.
+[](https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/activity?branch=main)
+[](https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/activity?branch=3.x)
+
+Our original Jenkins job (formerly `/Jenkinsfile`) is now
+`build-aux/Jenkinsfile.full`. This builds CouchDB on `main`, all of
+our release branches (`2.*`, `3.*`, etc.) as well as any branch prefixed with
+`jenkins-` for testing on a wide variety of supported operating systems.
Settings are as follows:
@@ -145,5 +114,18 @@ Settings are as follows:
* Max # of old items to keep: 10
* Everything else set as defaults.
+## Other Resources
+
+The [apache/couchdb-ci](https://github.com/apache/couchdb-ci) repo contains the
+dockerfiles that we use to generate the container images used for our
+container-based builds. These images are hosted on Docker Hub in the following
+repos:
+* [apache/couchdbci-debian](https://hub.docker.com/r/apache/couchdbci-debian)
+* [apache/couchdbci-ubuntu](https://hub.docker.com/r/apache/couchdbci-ubuntu)
+* [apache/couchdbci-centos](https://hub.docker.com/r/apache/couchdbci-centos)
+The [apache/couchdb-pkg](https://github.com/apache/couchdb-ci) repo contains
+a set of helper scripts to build binary packages for Debian / CentOS / Ubuntu
+from the contents of a release tarball. The packaging stage of our "Full
+Platform Builds" pipeline clones this repo to produces the package artifacts.
[couchdb] 03/09: Run same suites as 'check' target
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 9fd6b4514fbaffc564ee51aa8dc7dab304e2b253
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 14:54:11 2022 -0500
Run same suites as 'check' target
---
build-aux/Jenkinsfile.full | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 2315e03..c9dac11 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -126,8 +126,8 @@ def generateNativeStage(platform) {
sh '$MAKE'
sh '$MAKE eunit'
sh '$MAKE elixir-suite'
- sh '$MAKE exunit'
sh '$MAKE mango-test'
+ sh '$MAKE weatherreport-test'
}
}
}
@@ -178,8 +178,8 @@ def generateContainerStage(platform) {
sh 'make'
sh 'make eunit'
sh 'make elixir-suite'
- sh 'make exunit'
sh 'make mango-test'
+ sh 'make weatherreport-test'
}
}
catch (err) {
[couchdb] 07/09: Try to figure out why tree is dirty
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 7679624e393fed4f1ec7ec694d04e5ac45f2ffd4
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 21:40:47 2022 -0500
Try to figure out why tree is dirty
---
build-aux/Jenkinsfile.full | 1 +
1 file changed, 1 insertion(+)
diff --git a/build-aux/Jenkinsfile.full b/build-aux/Jenkinsfile.full
index 884951c..d1c0f19 100644
--- a/build-aux/Jenkinsfile.full
+++ b/build-aux/Jenkinsfile.full
@@ -276,6 +276,7 @@ pipeline {
steps {
timeout(time: 15, unit: "MINUTES") {
sh (script: 'rm -rf apache-couchdb-*', label: 'Clean workspace of any previous release artifacts' )
+ sh 'git status'
sh "./configure --spidermonkey-version ${spidermonkey}"
sh 'make erlfmt-check'
sh 'make elixir-check-formatted'
[couchdb] 06/09: Fix SM60 prohibition on ARM
Posted by ko...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kocolosk pushed a commit to branch jenkins-dynamic-matrix-3x
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 78a152f8b6ef9354e4be48d0567e22dcba58dd6b
Author: Adam Kocoloski <ko...@apache.org>
AuthorDate: Thu Jan 20 19:55:32 2022 -0500
Fix SM60 prohibition on ARM
Already fixed by @lostnet on main.
---
configure | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/configure b/configure
index 9c22181..0bcbfae 100755
--- a/configure
+++ b/configure
@@ -224,9 +224,9 @@ parse_opts() {
parse_opts $@
-if [ "${ARCH}" = "aarch64" ] && [ "${SM_VSN}" != "1.8.5" ]
+if [ "${ARCH}" = "aarch64" ] && [ "${SM_VSN}" = "60" ]
then
- echo "ERROR: SpiderMonkey 60 is known broken on ARM 64 (aarch64). Use 1.8.5 instead."
+ echo "ERROR: SpiderMonkey 60 is known broken on ARM 64 (aarch64). Use another version instead."
exit 1
fi