You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by jo...@apache.org on 2023/03/02 09:58:30 UTC
svn commit: r1907961 - in /httpd/httpd/branches/2.4.x: ./ .github/ .github/workflows/linux.yml .travis.yml test/README.ci test/README.travis test/travis_before_linux.sh test/travis_run_linux.sh
Author: jorton
Date: Thu Mar 2 09:58:30 2023
New Revision: 1907961
URL: http://svn.apache.org/viewvc?rev=1907961&view=rev
Log:
Merge r1891497, r1906480, r1906482, r1906501, r1907235, r1907235, r1907236, r1907237, r1907506, r1907506, r1907670, r1907960 from trunk:
* Allow to load /home/travis/build/apache/httpd/.gdbinit
Prep for GitHub actions: move CPAN handling into test scripts,
don't reference /home/travis directly.
Enable simplest CI test cases in GitHub Actions (currently also
still enabled in Travis). Based on apr/trunk.
Limit triggering on non-code changes, try to capture error_log as
an artifact on failure.
Migrate more tests to Github Actions.
Github: fixes #340, fixes #339
Migrate more tests to Github Actions.
Github: fixes #340, fixes #339
Travis RIP, WIP.
Add todo list for GHA migration.
Further migration of tests to GHA, though HTTP/2, ACME and
i386 build testing are all failing so currently disabled.
Caching needs adjusting to use a different model to Travis.
Further migration of tests to GHA, though HTTP/2, ACME and
i386 build testing are all failing so currently disabled.
Caching needs adjusting to use a different model to Travis.
Clear cache for mod_tls test.
Also run CI for PRs against 2.4.x.
Submitted by: rpluem, jorton
Added:
httpd/httpd/branches/2.4.x/.github/
- copied from r1906482, httpd/httpd/trunk/.github/
httpd/httpd/branches/2.4.x/test/README.ci
- copied, changed from r1907960, httpd/httpd/branches/2.4.x/test/README.travis
Removed:
httpd/httpd/branches/2.4.x/.travis.yml
httpd/httpd/branches/2.4.x/test/README.travis
Modified:
httpd/httpd/branches/2.4.x/ (props changed)
httpd/httpd/branches/2.4.x/.github/workflows/linux.yml
httpd/httpd/branches/2.4.x/test/travis_before_linux.sh
httpd/httpd/branches/2.4.x/test/travis_run_linux.sh
Propchange: httpd/httpd/branches/2.4.x/
------------------------------------------------------------------------------
Merged /httpd/httpd/trunk:r1891497,1906480,1906482,1906501,1907235-1907237,1907506,1907670,1907960
Modified: httpd/httpd/branches/2.4.x/.github/workflows/linux.yml
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/.github/workflows/linux.yml?rev=1907961&r1=1906482&r2=1907961&view=diff
==============================================================================
--- httpd/httpd/branches/2.4.x/.github/workflows/linux.yml (original)
+++ httpd/httpd/branches/2.4.x/.github/workflows/linux.yml Thu Mar 2 09:58:30 2023
@@ -3,8 +3,18 @@ name: Linux
on:
push:
branches: [ "*" ]
+ paths-ignore:
+ - 'docs/**'
+ - STATUS
+ - CHANGES
+ - changes-entries/*
pull_request:
- branches: [ "trunk" ]
+ branches: [ "trunk", "2.4.x" ]
+ paths-ignore:
+ - 'docs/**'
+ - STATUS
+ - CHANGES
+ - changes-entries/*
env:
MARGS: "-j2"
@@ -13,8 +23,15 @@ env:
jobs:
build:
strategy:
+ fail-fast: false
matrix:
include:
+ # -------------------------------------------------------------------------
+ - name: Empty APLOGNO() test
+ env: |
+ SKIP_TESTING=1
+ TEST_LOGNO=1
+ # -------------------------------------------------------------------------
- name: Default
# -------------------------------------------------------------------------
- name: All-static modules
@@ -34,13 +51,216 @@ jobs:
# -------------------------------------------------------------------------
- name: Event MPM, all-modules, no CMSG_DATA
config: --enable-mods-shared=reallyall --with-mpm=event ac_cv_have_decl_CMSG_DATA=no
- fail-fast: false
+ # -------------------------------------------------------------------------
+ - name: Default, all-modules + install
+ config: --enable-mods-shared=reallyall
+ env: |
+ TEST_INSTALL=1
+ APACHE_TEST_EXTRA_ARGS=-v
+ # -------------------------------------------------------------------------
+ - name: Default, all-modules, random test order
+ config: --enable-mods-shared=reallyall
+ env: |
+ TEST_ARGS=-order=random
+ # -------------------------------------------------------------------------
+ - name: GCC 10 maintainer-mode w/-Werror, install + VPATH
+ config: --enable-mods-shared=reallyall --enable-maintainer-mode
+ notest-cflags: -Werror -O2 -Wno-deprecated-declarations
+ env: |
+ CC=gcc-10
+ TEST_VPATH=1
+ TEST_INSTALL=1
+ SKIP_TESTING=1
+ # -------------------------------------------------------------------------
+ - name: All-modules, APR 1.7.2, APR-util 1.6.3
+ config: --enable-mods-shared=reallyall
+ env: |
+ APR_VERSION=1.7.2
+ APU_VERSION=1.6.3
+ APU_CONFIG="--with-crypto --with-ldap"
+ # -------------------------------------------------------------------------
+ - name: APR 1.7.x, APR-util 1.7.x
+ config: --enable-mods-shared=reallyall
+ env: |
+ APR_VERSION=1.7.x
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: Pool-debug
+ config: --enable-mods-shared=reallyall
+ env: |
+ APR_VERSION=1.7.x
+ APR_CONFIG="--enable-pool-debug"
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_MALLOC=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: Shared MPMs (event), pool-debug, SSL/TLS variants
+ config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=event
+ env: |
+ APR_VERSION=1.7.x
+ APR_CONFIG="--enable-pool-debug"
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_MALLOC=1
+ TEST_SSL=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: Shared MPMs (worker), pool-debug, SSL/TLS variants
+ config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=worker
+ env: |
+ APR_VERSION=1.7.x
+ APR_CONFIG="--enable-pool-debug"
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_MALLOC=1
+ TEST_SSL=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: Shared MPMs (prefork), pool-debug, SSL/TLS variants
+ config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=prefork
+ env: |
+ APR_VERSION=1.7.x
+ APR_CONFIG="--enable-pool-debug"
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_MALLOC=1
+ TEST_SSL=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: litmus WebDAV tests
+ config: --enable-dav --enable-dav-fs
+ env: |
+ LITMUS=1
+ TESTS="t/modules/dav.t"
+ pkgs: litmus
+ # -------------------------------------------------------------------------
+ # MFLAGS= works around https://bz.apache.org/bugzilla/show_bug.cgi?id=63942
+ ## TODO if: *condition_not_24x
+ - name: Regenerate ap_expr
+ config: --enable-mods-shared=reallyall --enable-maintainer-mode
+ notest-cflags: -Werror -Wno-deprecated-declarations
+ env: |
+ BUILDCONFIG="--with-regen-expr"
+ MFLAGS=
+ # -------------------------------------------------------------------------
+ - name: APR 1.7.2, APR-util 1.6.3, LDAP
+ config: --enable-mods-shared=reallyall
+ pkgs: ldap-utils
+ env: |
+ APR_VERSION=1.7.2
+ APU_VERSION=1.6.3
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_MALLOC=1
+ TEST_LDAP=1
+ TEST_ARGS="-defines LDAP"
+ TESTS="t/modules/"
+ # -------------------------------------------------------------------------
+ ### TODO: if: *condition_not_24x
+ - name: APR trunk thread debugging
+ config: --enable-mods-shared=reallyall --with-mpm=event
+ env: |
+ APR_VERSION=trunk
+ APR_CONFIG="--with-crypto --enable-thread-debug"
+ # -------------------------------------------------------------------------
+ - name: UBSan
+ notest-cflags: -fsanitize=undefined -fno-sanitize-recover=undefined
+ config: --enable-mods-shared=reallyall --disable-http2
+ env: |
+ NOTEST_LIBS=-lubsan
+ TEST_UBSAN=1
+ # -------------------------------------------------------------------------
+ - name: ASan
+ notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer
+ config: --enable-mods-shared=reallyall
+ env: |
+ APR_VERSION=1.7.x
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_ASAN=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ - name: ASan, pool-debug
+ notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer
+ config: --enable-mods-shared=reallyall
+ env: |
+ APR_VERSION=1.7.x
+ APR_CONFIG="--enable-pool-debug"
+ APU_VERSION=1.7.x
+ APU_CONFIG="--with-crypto --with-ldap"
+ TEST_ASAN=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ ### TODO: if: *condition_not_24x
+ ### TODO: fails in tests.
+ # - name: HTTP/2 test suite
+ # config: --enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=all
+ # pkgs: curl python3-pytest nghttp2-client python3-cryptography python3-requests
+ # env: |
+ # APR_VERSION=1.7.2
+ # APU_VERSION=1.6.1
+ # APU_CONFIG="--with-crypto"
+ # NO_TEST_FRAMEWORK=1
+ # TEST_INSTALL=1
+ # TEST_H2=1
+ # TEST_CORE=1
+ # TEST_PROXY=1
+ # -------------------------------------------------------------------------
+ ### TODO: if: *condition_not_24x
+ ### TODO: pebble install is broken.
+ # - name: ACME test suite
+ # config: --enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=event
+ # pkgs: >-
+ # python3-pytest nghttp2-client python3-cryptography python3-requests python3-filelock
+ # golang-1.17 curl
+ # env: |
+ # APR_VERSION=1.7.2
+ # APU_VERSION=1.6.1
+ # APU_CONFIG="--with-crypto"
+ # GOROOT=/usr/lib/go-1.17
+ # NO_TEST_FRAMEWORK=1
+ # TEST_INSTALL=1
+ # TEST_MD=1
+ # -------------------------------------------------------------------------
+ ### TODO: if: *condition_not_24x
+ ### TODO: fix caching here.
+ - name: MOD_TLS test suite
+ config: --enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=event
+ pkgs: curl python3-pytest nghttp2-client python3-cryptography python3-requests cargo cbindgen
+ env: |
+ APR_VERSION=1.7.2
+ APU_VERSION=1.6.1
+ APU_CONFIG="--with-crypto"
+ NO_TEST_FRAMEWORK=1
+ TEST_INSTALL=1
+ TEST_MOD_TLS=1
+ CLEAR_CACHE=1
+ # -------------------------------------------------------------------------
+ ### TODO if: *condition_not_24x
+ ### TODO: Fails because :i386 packages are not being found.
+ # - name: i386 Shared MPMs, most modules, maintainer-mode w/-Werror
+ # config: --enable-mods-shared=reallyall --disable-xml2enc --disable-proxy-html --enable-mpms-shared=all --enable-maintainer-mode
+ # pkgs: >-
+ # cpanminus libc6-dev-i386 gcc-multilib libexpat1-dev:i386 libssl-dev:i386
+ # lib32z1-dev libbrotli-dev:i386 libpcre2-dev:i386 libldap2-dev:i386 libtool-bin
+ # perl-doc libapr1-dev libbrotli-dev:i386
+ # env: |
+ # PKG_CONFIG_PATH="/usr/lib/i386-linux-gnu/pkgconfig"
+ # NOTEST_CFLAGS="-Werror"
+ # CC="gcc -m32"
+ # APR_VERSION=1.7.0
+ # APU_VERSION=1.6.1
+ # APU_CONFIG="--with-crypto --with-ldap"
runs-on: ubuntu-latest
env:
NOTEST_CFLAGS: ${{ matrix.notest-cflags }}
CONFIG: ${{ matrix.config }}
name: ${{ matrix.name }}
steps:
+ - name: Set environment variables
+ run: echo "${{ matrix.env }}" >> $GITHUB_ENV
- name: apt refresh
run: sudo apt-get -o Acquire::Retries=5 update
- name: Install prerequisites
@@ -48,9 +268,36 @@ jobs:
cpanminus libtool-bin libapr1-dev libaprutil1-dev
liblua5.3-dev libbrotli-dev libcurl4-openssl-dev
libnghttp2-dev libjansson-dev libpcre2-dev gdb
- perl-doc
+ perl-doc ${{ matrix.pkgs }}
- uses: actions/checkout@v3
+ - name: Cache APR build
+ uses: actions/cache@v3
+ env:
+ cache-name: cache-apru
+ with:
+ path: ~/root
+ key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('/home/runner/root/.key-*') }}
+ - name: Cache CPAN modules
+ uses: actions/cache@v3
+ env:
+ cache-name: cache-cpan
+ with:
+ path: ~/perl5
+ key: ${{ runner.os }}-cpan-${{ env.cache-name }}-${{ hashFiles('/home/runner/perl5/.key') }}
- name: Configure environment
run: ./test/travis_before_linux.sh
+ timeout-minutes: 15
+ - uses: actions/upload-artifact@v3
+ if: failure()
+ with:
+ name: config.log ${{ matrix.node-version }}
+ path: |
+ /home/runner/build/**/config.log
- name: Build and test
run: ./test/travis_run_linux.sh
+ - uses: actions/upload-artifact@v3
+ if: failure()
+ with:
+ name: error_log ${{ matrix.node-version }}
+ path: test/perl-framework/t/logs/error_log
+
Copied: httpd/httpd/branches/2.4.x/test/README.ci (from r1907960, httpd/httpd/branches/2.4.x/test/README.travis)
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/test/README.ci?p2=httpd/httpd/branches/2.4.x/test/README.ci&p1=httpd/httpd/branches/2.4.x/test/README.travis&r1=1907960&r2=1907961&rev=1907961&view=diff
==============================================================================
--- httpd/httpd/branches/2.4.x/test/README.travis (original)
+++ httpd/httpd/branches/2.4.x/test/README.ci Thu Mar 2 09:58:30 2023
@@ -2,7 +2,7 @@
Variables
---------
-The Travis scripts use the following environment variables:
+The CI scripts use the following environment variables:
* APR_VERSION - if set, APR of this version is built and installed in
$HOME/root/apr-$APR_VERSION - a value of "trunk" means trunk is
@@ -44,7 +44,9 @@ The Travis scripts use the following env
* TEST_ARGS - arguments to pass to ./t/TEST in the Perl test framework
-Caching
+* CLEAR_CACHE - if set, the cached $HOME/root is removed before each build
+
+Caching -- NOTE, BROKEN IN GITHUB ACTIONS --
-------
Perl modules installed in $HOME/perl5 are cached.
@@ -53,27 +55,72 @@ Anything installed into the $HOME/root d
versions of APR/APR-util are installed here and cached across httpd
build jobs without needing to be rebuilt every time.
+The cached installs of APR/APR-util are refreshed if the
+last-changed-revision of the build is stale.
+
+If APR_VERSION and APU_VERSION are both set to 1.x versions, then
+CLEAR_CACHE should also be set to disable APR* caching. APR-util can
+only be rebuilt if an APR checkout is present, so a APR-util cannot be
+built from source alone. (i.e. the scripts do not handle the case of
+cached, fresh APR plus a cached but stale APR-util)
+
+Travis to Github Actions Migration TODO
+---------------------------------------
+
+* better path filtering so e.g. CHANGES changes don't trigger CI
+* support branch conditionals again (some tests are 2.4.x only, some trunk only)
+* make caching work properly for APR + CPAN modules
+ - this is using the wrong model at the moment
+ - the cache key needs to be based off (source code, job configuration)
+ - rather than done on the fly in test/travis_before_linux.sh
+ - pebble + Rustls builds should also be cached
+* turn on failure notifications?
+* test across different Ubuntu versions again
+ - and test against OpenSSL 1.x since we're now ONLY building against 3.x
+* update the docs below for testing from PRs/feature branches
+* introduce some job ordering rather than having a flat/concurrent
+ set, if the default "./configure && make && test" works *then* start
+ jobs doing 200 different variations on ./configure --enable-XXX
+ i.e. stop burning CPU time for a typo which breaks every job
+
TODO list
---------
+* non-x86 builds
* MacOS build
* Windows build
* clang-on-Linux build
* Use containers for non-Ubuntu-based Linux testing
-* sanity checks for use of APLOGNO() - empty arguments, accidental duplicates, etc.
- - not sure how exactly
-* Known test failures
- - "apt-get install" timeout/fails - workaround by moving apt install to before_script phase?
-Testing from a Feature Branch
+Known Failures
+--------------
+
+Some known failures:
+
+* prefork, and more rarely non-prefork testing sometimes catches child
+ segfaults under pool-debug from assert()s in (e.g.)
+ __pthread_tpp_change_priority where one child is destroying threads
+ which another is waiting for, or iterating through via
+ apr_pool_walk_tree().
+
+ See dev@httpd threads:
+ msg <5f...@travis-tasks-5b566d48fc-drkb9.mail>
+ msg <73...@kippdata.de> and
+ https://bz.apache.org/bugzilla/show_bug.cgi?id=63098
+ https://bz.apache.org/bugzilla/show_bug.cgi?id=46185
+
+ Not clear if there is a real bug here which can be reproduced
+ outside of pool-debug.
+
+Testing from a Feature Branch [*** N/A FOR GITHUB ACTIONS ***]
-----------------------------
An SVN branch off trunk should be mirrored to github, and will be
-tested in the same way that trunk is in Travis, so this workflow is
+tested in the same way that trunk is in CI, so this workflow is
available for those familiar with using Subversion and the standard
ASF/httpd repository layout.
-Tested branches are listed at: https://travis-ci.org/github/apache/httpd/branches
+Tested branches are listed at: https://travis-ci.com/github/apache/httpd/branches
Travis will also run the tests for a PR filed against the httpd Github
repository at https://github.com/apache/httpd or from a fork of this
@@ -91,16 +138,17 @@ $USERNAME for your github username:
$ git push -u $USERNAME my-feature:my-feature
To enable testing for a fork, visit the settings page at
-https://travis-ci.org/$USERNAME/httpd/settings - you may need to sync
-your account via https://travis-ci.org/account/repositories for a
+https://travis-ci.com/$USERNAME/httpd/settings - you may need to sync
+your account via https://travis-ci.com/account/repositories for a
freshly created fork.
-To create a Pull Request, go to a URL like:
+To create a Pull Request, go to the URL produced in the "git push"
+command output when pushing to your fork, which is something like:
https://github.com/apache/httpd/compare/trunk...$USERNAME:trunk
Once a PR has been created, travis will run the tests and link the
results from a PR comment. All tested PRs are listed here:
-https://travis-ci.org/github/apache/httpd/pull_requests
+https://travis-ci.com/github/apache/httpd/pull_requests
To merge from github back to SVN trunk, create a patch from e.g.:
Modified: httpd/httpd/branches/2.4.x/test/travis_before_linux.sh
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/test/travis_before_linux.sh?rev=1907961&r1=1907960&r2=1907961&view=diff
==============================================================================
--- httpd/httpd/branches/2.4.x/test/travis_before_linux.sh (original)
+++ httpd/httpd/branches/2.4.x/test/travis_before_linux.sh Thu Mar 2 09:58:30 2023
@@ -68,9 +68,9 @@ function install_apx() {
local revision=`svn info --show-item last-changed-revision ${url}`
- # Blow away the cached install root if the revision does not
- # match.
- test -f ${prefix}/.revision-is-${revision} || rm -rf ${prefix}
+ # Blow away the cached install root if the cached install is stale
+ # or doesn't match the expected configuration.
+ grep -q "${version} ${revision} ${config} CC=$CC" ${HOME}/root/.key-${name} || rm -rf ${prefix}
if test -d ${prefix}; then
return 0
@@ -84,13 +84,33 @@ function install_apx() {
make install
popd
- touch ${prefix}/.revision-is-${revision}
+ echo ${version} ${revision} "${config}" "CC=${CC}" > ${HOME}/root/.key-${name}
}
+# Allow to load $HOME/build/apache/httpd/.gdbinit
+echo "add-auto-load-safe-path $HOME/build/apache/httpd/.gdbinit" >> $HOME/.gdbinit
+# Prepare perl-framework test environment
if ! test -v SKIP_TESTING; then
- ### Temporary: purge old svn checkout from the cache
- rm -rf $HOME/root/framework
+ # Clear CPAN cache if necessary
+ if [ -v CLEAR_CACHE ]; then rm -rf ~/perl5; fi
+
+ cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
+
+ pkgs="Net::SSL LWP::Protocol::https \
+ LWP::Protocol::AnyEvent::http ExtUtils::Embed Test::More \
+ AnyEvent DateTime HTTP::DAV FCGI \
+ AnyEvent::WebSocket::Client Apache::Test"
+
+ # CPAN modules are to be used with the system Perl and always with
+ # CC=gcc, e.g. for the CC="gcc -m32" case the builds are not correct
+ # otherwise.
+ CC=gcc cpanm --notest $pkgs
+
+ # Set cache key.
+ echo $pkgs > ~/perl5/.key
+ unset pkgs
+
# Make a shallow clone of httpd-tests git repo.
git clone --depth=1 https://github.com/apache/httpd-tests.git test/perl-framework
fi
Modified: httpd/httpd/branches/2.4.x/test/travis_run_linux.sh
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/test/travis_run_linux.sh?rev=1907961&r1=1907960&r2=1907961&view=diff
==============================================================================
--- httpd/httpd/branches/2.4.x/test/travis_run_linux.sh (original)
+++ httpd/httpd/branches/2.4.x/test/travis_run_linux.sh Thu Mar 2 09:58:30 2023
@@ -39,6 +39,9 @@ if test ! -v SKIP_TESTING; then
CONFIG="--with-test-suite=test/perl-framework $CONFIG"
WITH_TEST_SUITE=1
fi
+
+ # Use the CPAN environment.
+ eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
fi
if test -v APR_VERSION; then
CONFIG="$CONFIG --with-apr=$HOME/root/apr-${APR_VERSION}"
@@ -191,7 +194,6 @@ if ! test -v SKIP_TESTING; then
# Run ACME tests.
# need the go based pebble as ACME test server
# which is a package on debian sid, but not on focal
- export GOROOT=/usr/lib/go-1.14
export GOPATH=${PREFIX}/gocode
mkdir -p "${GOPATH}"
export PATH="${GOROOT}/bin:${GOPATH}/bin:${PATH}"