You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by jo...@apache.org on 2026/08/27 06:31:26 UTC
svn commit: r1937498 - in httpd/httpd/trunk: .github/workflows test
Author: jorton
Date: Thu Aug 27 06:31:26 2026
New Revision: 1937498
Log:
CI: Key the ~/root cache on the dependencies, retiring CLEAR_CACHE.
Caches in GHA are immutable, so a key which does not change when a
dependency does can never be refreshed; branch builds were therefore
rebuilt on every run, and CLEAR_CACHE disabled caching altogether for
the nine jobs using them. The key also hashed the whole matrix entry,
so jobs building identical dependencies each had their own copy.
* test/gha-resolve-deps.sh: Added. Compute $ROOTID from the dependency
versions, their configure arguments and the compiler flags, resolving
any branch to a commit; export the commit so that the build uses what
the key names rather than resolving it a second time.
* .github/workflows/linux.yml: Key ~/root on $ROOTID, skip the cache
where nothing is built from source, and save it only once the
dependencies have built successfully. Drop CLEAR_CACHE=1 from the
nine jobs which set it.
* test/travis_before_linux.sh (resolve_apx, fetch_apx): Split out of
install_apx().
(install_apx): Fetch the APR source before building APR-util if APR
came from the cache; trust a restored install root. Drop the
CLEAR_CACHE handling and the .key-* and openssl-is-* staleness files,
which the cache key now covers.
* test/README.ci: Update accordingly.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
GitHub: PR #717
Added:
httpd/httpd/trunk/test/gha-resolve-deps.sh (contents, props changed)
Modified:
httpd/httpd/trunk/.github/workflows/linux.yml
httpd/httpd/trunk/test/README.ci
httpd/httpd/trunk/test/travis_before_linux.sh
Modified: httpd/httpd/trunk/.github/workflows/linux.yml
==============================================================================
--- httpd/httpd/trunk/.github/workflows/linux.yml Thu Aug 27 06:31:05 2026 (r1937497)
+++ httpd/httpd/trunk/.github/workflows/linux.yml Thu Aug 27 06:31:26 2026 (r1937498)
@@ -117,7 +117,6 @@ jobs:
APR_VERSION=1.7.x
APU_VERSION=1.6.x
APU_CONFIG="--with-crypto --with-ldap"
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR 1.8.x, APR-util 1.7.x
config: --enable-mods-shared=reallyall
@@ -125,7 +124,6 @@ jobs:
APR_VERSION=1.8.x
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR 1.7.x, APR-util 1.7.x, LDAP
config: --enable-mods-shared=reallyall
@@ -139,7 +137,6 @@ jobs:
TEST_ARGS="-defines LDAP"
TESTS="t/modules/"
TEST_PYTEST=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR trunk thread debugging
config: --enable-mods-shared=reallyall --with-mpm=event
@@ -155,7 +152,6 @@ jobs:
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (event), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=event
@@ -166,7 +162,6 @@ jobs:
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (worker), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=worker
@@ -177,7 +172,6 @@ jobs:
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (prefork), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=prefork
@@ -188,7 +182,6 @@ jobs:
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: litmus WebDAV tests
config: --enable-dav --enable-dav-fs
@@ -233,7 +226,6 @@ jobs:
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
TEST_PYTEST=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: ASan, pool-debug
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer -O2
@@ -246,7 +238,6 @@ jobs:
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
TEST_PYTEST=1
- CLEAR_CACHE=1
# -------------------------------------------------------------------------
# Runs every pytest-based test suite (pytest_suite/ + all
# test/modules/*/ pyhttpd suites except modules/md, which needs a
@@ -368,11 +359,22 @@ jobs:
path: test/perl-framework
submodules: true
if: ${{ ! ( env.SKIP_TESTING || env.NO_TEST_FRAMEWORK ) }}
- - name: Cache installed libraries
- uses: actions/cache@v5
+ # Sets $ROOTID, identifying the contents of ~/root, and leaves it
+ # unset for a job which builds no dependencies from source.
+ - name: Resolve dependency versions
+ run: ./test/gha-resolve-deps.sh
+ # Restored here but saved only once the dependencies have been built
+ # successfully, below: a job which fails part-way through building
+ # them would otherwise save a half-built tree under a key which can
+ # never be updated. Bump the serial in the key to discard the cached
+ # builds and start afresh.
+ - name: Restore installed libraries
+ uses: actions/cache/restore@v5
+ id: cache-libs
+ if: env.ROOTID != ''
with:
path: ~/root
- key: cache-libs-${{ env.JOBID }}
+ key: cache-libs-1-${{ env.IMAGEOS }}-${{ runner.arch }}-${{ env.ROOTID }}
# ~/perl5 does not vary by job: the module list is fixed and the
# modules are always built with CC=gcc. It varies only by the perl
# ABI (hence the image and architecture) and by the module list
@@ -387,6 +389,12 @@ jobs:
- name: Configure environment
run: ./test/travis_before_linux.sh
timeout-minutes: 15
+ - name: Save installed libraries
+ uses: actions/cache/save@v5
+ if: success() && env.ROOTID != '' && steps.cache-libs.outputs.cache-hit != 'true'
+ with:
+ path: ~/root
+ key: ${{ steps.cache-libs.outputs.cache-primary-key }}
- uses: actions/upload-artifact@v7
if: failure()
with:
Modified: httpd/httpd/trunk/test/README.ci
==============================================================================
--- httpd/httpd/trunk/test/README.ci Thu Aug 27 06:31:05 2026 (r1937497)
+++ httpd/httpd/trunk/test/README.ci Thu Aug 27 06:31:26 2026 (r1937498)
@@ -99,9 +99,6 @@ The CI scripts use the following environ
included in the pytest test suites. Set globally in the workflow
(e.g. "/usr/sbin/php-fpm8.3").
-* CLEAR_CACHE - if set, the cached $HOME/root and $HOME/perl5 are
- removed before each build
-
Caching
-------
@@ -119,26 +116,34 @@ key, not in the script.
Two directories are cached.
-~/root (key "cache-libs-$JOBID") holds the APR, APR-util and OpenSSL
-installs. $JOBID is a hash of the runner image ($ImageOS), the
-architecture ($RUNNER_ARCH), and the matrix entry's notest-cflags, env
-and config. Hence if e.g. APR_VERSION or APR_CONFIG changes, a
-different cache key will be used for future builds, but while it
-remains the same the cached install can be used across builds.
-
-This does not work optimally for e.g. APR_VERSION=trunk - in this
-case, a trunk build of APR is built and cached (assuming the job
-succeeds). Once the trunk revision changes, install_apx() will see the
-cached trunk install is stale and throw it away for every single
-subsequent job; the cache is never updated.
+~/root (key "cache-libs-<serial>-<image>-<arch>-$ROOTID") holds the
+APR, APR-util and OpenSSL installs. gha-resolve-deps.sh computes
+$ROOTID before the cache is restored, hashing only what determines
+those installs: the versions, their configure arguments and $CC. The
+job's own configuration is deliberately not included, so jobs building
+the same dependencies share one entry however differently they
+configure or test httpd - the five pool-debug jobs, for instance. A job
+which builds no dependencies gets no $ROOTID and does not use the cache
+at all.
+
+For anything built from a branch rather than a tag - APR_VERSION=trunk
+or 1.7.x, and TEST_OPENSSL3_BRANCH - the version alone does not identify
+what gets built, so the branch is resolved to a commit and that goes
+into $ROOTID too. The commit is exported as
+$APR_COMMIT/$APU_COMMIT/$OPENSSL_COMMIT so that the build uses exactly
+what the key names rather than resolving the branch a second time,
+minutes later, by which point it may have moved.
+
+It is restored before the dependencies are built and saved afterwards,
+only if they built successfully - a failed job would otherwise write a
+half-built tree to a key which can never be updated.
~/perl5 (key "cache-cpan-<serial>-<image>-<arch>-<hash>") holds the
CPAN modules. Nothing in it varies per job: the module list is fixed
and the modules are always built with CC=gcc, so the content depends
only on the perl ABI - which is determined by the image and the
architecture - and on the module list itself, hashed from
-travis_before_linux.sh. One cache is therefore shared by every job
-rather than one per $JOBID. The .perlver check in the script remains
+travis_before_linux.sh. One cache is therefore shared by every job. The .perlver check in the script remains
as a backstop for a perl upgrade within the same image.
The cached CPAN modules still become gradually stale as versions in
@@ -146,29 +151,14 @@ CPAN move on from whatever was installed
the serial in the key ("cache-cpan-1-" -> "cache-cpan-2-") to discard
them and rebuild against current CPAN.
-If APR_VERSION and APU_VERSION are both set to 1.x versions, then
-CLEAR_CACHE should also be set to disable APR* caching. APR-util can
-only be rebuilt if an APR checkout is present, so a APR-util cannot be
-built from source alone. (i.e. the scripts do not handle the case of
-cached, fresh APR plus a cached but stale APR-util)
-
-Still to do:
-
-* Put the resolved commit hash in the cache key for APR/APR-util and
- OpenSSL branch builds, as in the example documentation:
- https://github.com/actions/cache/blob/main/examples.md
- That would retire CLEAR_CACHE along with the hand-rolled staleness
- logic - the .key-* files in install_apx() and the openssl-is-*
- stamps, which are two independent implementations of the same
- "remote HEAD vs recorded HEAD" check. Note APR and APR-util must
- stay in one cache entry, for the reason given above.
-* Share ~/root between jobs which build identical dependencies; the
- five pool-debug jobs each build the same APR 1.7.x + APR-util 1.7.x
- into a separate cache key.
-* Use actions/cache/restore plus actions/cache/save with
- if: success(), so that a job which fails part-way through building a
- dependency cannot write a broken tree to a key which can never be
- updated.
+APR and APR-util must stay in a single cache entry. APR-util's
+buildconf needs APR's *source* tree, which lives in $HOME/build and is
+not cached, so a cached but stale APR-util alongside a cached and fresh
+APR cannot be rebuilt from the cache alone. This used to be handled by
+setting CLEAR_CACHE on such jobs to disable caching altogether;
+install_apx() now fetches the APR source on demand in that case
+instead, so no job needs to opt out of caching.
+
Travis to Github Actions Migration TODO
---------------------------------------
Added: httpd/httpd/trunk/test/gha-resolve-deps.sh
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ httpd/httpd/trunk/test/gha-resolve-deps.sh Thu Aug 27 06:31:26 2026 (r1937498)
@@ -0,0 +1,84 @@
+#!/bin/bash -e
+#
+# Work out what identifies the contents of ~/root, and write it to
+# $GITHUB_ENV as $ROOTID for the cache key to use.
+#
+# ~/root holds only the dependencies built from source, so the key
+# covers those alone - not the job's own configuration. Jobs which
+# build the same dependencies then share one cache entry even where they
+# configure or test httpd differently. $ROOTID is left unset for a job
+# which builds no dependencies at all, and the cache is skipped.
+#
+# Anything built from a branch is resolved to a commit here and the
+# commit exported, so that the build uses exactly what the key names;
+# resolving the branch again at build time would race with it moving.
+# Tags and release tarballs are immutable, so the version is enough.
+
+root=
+
+# $1 = name, $2 = version, $3 = configure arguments, $4 = variable to
+# export the resolved commit into.
+function add_apx() {
+ local name=$1
+ local version=$2
+ local config=$3
+ local var=$4
+ local sha
+
+ if test -z "$version"; then
+ return 0
+ fi
+
+ case "$version" in
+ trunk|*.x)
+ sha=`git ls-remote https://github.com/apache/${name}.git refs/heads/${version} | cut -f1`
+ if test -z "$sha"; then
+ : Could not resolve ${name} branch ${version} - check branch is valid?
+ exit 1
+ fi
+ echo "${var}=${sha}" >> $GITHUB_ENV ;;
+ *) sha=${version} ;;
+ esac
+
+ root="${root} ${name}-${version}-${sha}-${config}"
+}
+
+add_apx apr "${APR_VERSION-}" "${APR_CONFIG-}" APR_COMMIT
+add_apx apr-util "${APU_VERSION-}" "${APU_CONFIG-}" APU_COMMIT
+
+if test -n "${TEST_OPENSSL3-}"; then
+ if test -n "${TEST_OPENSSL3_BRANCH-}"; then
+ sha=`git ls-remote https://github.com/openssl/openssl refs/heads/${TEST_OPENSSL3_BRANCH} | cut -f1`
+ if test -z "$sha"; then
+ : Could not resolve openssl branch ${TEST_OPENSSL3_BRANCH} - check branch is valid?
+ exit 1
+ fi
+ echo "OPENSSL_COMMIT=${sha}" >> $GITHUB_ENV
+ root="${root} openssl-${TEST_OPENSSL3_BRANCH}-${sha}-${OPENSSL_CONFIG-}"
+ else
+ root="${root} openssl-${TEST_OPENSSL3}-${OPENSSL_CONFIG-}"
+ fi
+fi
+
+if test -n "${TEST_NGHTTP2-}"; then
+ root="${root} nghttp2-${TEST_NGHTTP2}"
+fi
+
+if test -n "${RUSTLS_VERSION-}"; then
+ root="${root} rustls-${RUSTLS_VERSION}"
+fi
+
+if test -n "$root"; then
+ # The compiler and every flag which reaches the build are baked into
+ # the libraries, NOTEST_* included: APR adds those to the build too,
+ # it just does not use them for ./configure.
+ root="${root} CC=${CC-} CFLAGS=${CFLAGS-} CPPFLAGS=${CPPFLAGS-}"
+ root="${root} LDFLAGS=${LDFLAGS-} LIBS=${LIBS-}"
+ root="${root} NOTEST_CFLAGS=${NOTEST_CFLAGS-} NOTEST_CPPFLAGS=${NOTEST_CPPFLAGS-}"
+ root="${root} NOTEST_LDFLAGS=${NOTEST_LDFLAGS-} NOTEST_LIBS=${NOTEST_LIBS-}"
+ root="${root}${TEST_APR_TARBALL+ tarball}"
+ : Dependencies:${root}
+ echo ROOTID=`echo "$root" | md5sum - | sed 's/ .*//'` >> $GITHUB_ENV
+else
+ : No dependencies are built from source, so ~/root is not cached.
+fi
Modified: httpd/httpd/trunk/test/travis_before_linux.sh
==============================================================================
--- httpd/httpd/trunk/test/travis_before_linux.sh Thu Aug 27 06:31:05 2026 (r1937497)
+++ httpd/httpd/trunk/test/travis_before_linux.sh Thu Aug 27 06:31:26 2026 (r1937498)
@@ -1,9 +1,5 @@
#!/bin/bash -xe
-if test -v CLEAR_CACHE; then
- rm -rf $HOME/root
-fi
-
: Travis tag = ${TRAVIS_TAG}
: Travis branch = ${TRAVIS_BRANCH}
@@ -21,61 +17,111 @@ if grep ip6-localhost /etc/hosts; then
cat /etc/hosts
fi
-function install_apx() {
+# Echo the object ID (hash) of the commit to build for $1 at version
+# $2, or "tarball" if a release tarball is used instead of a checkout.
+function resolve_apx() {
local name=$1
local version=$2
- local prefix=${HOME}/root/${name}-${version}
- local build=${HOME}/build/${name}-${version}
- local giturl=https://github.com/apache/${name}.git
- local config=$3
- local buildconf=$4
- local commit=tarball
+ local ref commit
- mkdir -p ${HOME}/build
+ if test -v TEST_APR_TARBALL; then
+ echo tarball
+ return 0
+ fi
- if ! test -v TEST_APR_TARBALL; then
- case $version in
- trunk|*.x) ref=refs/heads/${version} ;;
- *) ref=refs/tags/${version} ;;
- esac
-
- # Fetch the object ID (hash) of latest commit
- commit=`git ls-remote ${giturl} ${ref} | cut -f1`
- if test -z "$commit"; then
- : Could not determine latest commit hash for ${ref} in ${giturl} - check branch is valid?
- exit 1
- fi
- fi
-
- # Blow away the cached install root if the cached install is stale
- # or doesn't match the expected configuration.
- grep -q "${version} ${commit} ${config} CC=$CC" ${HOME}/root/.key-${name} || rm -rf ${prefix}
+ # For a branch, prefer the commit already resolved by
+ # gha-resolve-deps.sh for the cache key; resolving it again here
+ # would race with the branch moving in the meantime. Tags are
+ # immutable so they are always resolved below.
+ case $name in
+ apr) commit=${APR_COMMIT-} ;;
+ apr-util) commit=${APU_COMMIT-} ;;
+ esac
+ if test -n "$commit"; then
+ echo ${commit}
+ return 0
+ fi
- if test -d ${prefix}; then
+ case $version in
+ trunk|*.x) ref=refs/heads/${version} ;;
+ *) ref=refs/tags/${version} ;;
+ esac
+
+ commit=`git ls-remote https://github.com/apache/${name}.git ${ref} | cut -f1`
+ if test -z "$commit"; then
+ : Could not determine latest commit hash for ${ref} in ${name} - check branch is valid?
+ exit 1
+ fi
+
+ echo ${commit}
+}
+
+# Unpack the source for $1 at version $2, commit $3, into
+# $HOME/build/$1-$2, running ./buildconf with the arguments in $4. Does
+# nothing if the source tree is already present. Note $HOME/build is
+# not cached, unlike the install root in $HOME/root.
+function fetch_apx() {
+ local name=$1
+ local version=$2
+ local commit=$3
+ local buildconf=$4
+ local build=${HOME}/build/${name}-${version}
+
+ if test -d ${build}; then
return 0
fi
+ mkdir -p ${HOME}/build
+
if test -v TEST_APR_TARBALL; then
curl https://archive.apache.org/dist/apr/${name}-${version}.tar.gz > apx.tar.gz
tar -C ${HOME}/build -xzf apx.tar.gz
rm apx.tar.gz
- pushd ${build}
else
git init -q ${build}
pushd $build
# Clone and checkout the commit identified above.
- git remote add origin ${giturl}
+ git remote add origin https://github.com/apache/${name}.git
git fetch -q --depth=1 origin ${commit}
git checkout ${commit}
./buildconf ${buildconf}
+ popd
fi
+}
+
+function install_apx() {
+ local name=$1
+ local version=$2
+ local prefix=${HOME}/root/${name}-${version}
+ local build=${HOME}/build/${name}-${version}
+ local config=$3
+ local buildconf=$4
+ local commit
+
+ # The cache key covers the version, the resolved commit for a branch
+ # build, the configuration and $CC, and is only written once the
+ # build has succeeded, so anything restored here is usable as-is.
+ if test -d ${prefix}; then
+ return 0
+ fi
+
+ commit=`resolve_apx ${name} ${version}`
+
+ # apr-util's buildconf needs APR's source tree, which is not cached.
+ # If APR itself was restored from the cache it was never built here,
+ # so fetch the source now - otherwise a stale APR-util alongside a
+ # fresh APR cannot be rebuilt.
+ if test ${name} = apr-util -a ! -d ${HOME}/build/apr-${APR_VERSION}; then
+ fetch_apx apr ${APR_VERSION} `resolve_apx apr ${APR_VERSION}`
+ fi
+
+ fetch_apx ${name} ${version} ${commit} "${buildconf}"
+ pushd ${build}
./configure --prefix=${prefix} ${config}
make -j2
make install
popd
-
- echo ${version} ${commit} "${config}" "CC=${CC}" > ${HOME}/root/.key-${name}
}
# Allow to load $HOME/build/apache/httpd/.gdbinit
@@ -84,9 +130,6 @@ echo "add-auto-load-safe-path $HOME/work
# Unless either SKIP_TESTING or NO_TEST_FRAMEWORK are set, install
# CPAN modules required to run the Perl test framework.
if ! test -v SKIP_TESTING -o -v NO_TEST_FRAMEWORK; then
- # Clear CPAN cache if necessary
- if [ -v CLEAR_CACHE ]; then rm -rf ~/perl5; fi
-
if ! perl -V > perlver; then
: Perl binary broken
perl -V
@@ -138,26 +181,20 @@ fi
# Build the requested version of OpenSSL if it's not already installed
# in the cached ~/root
if test -v TEST_OPENSSL3; then
- # For a branch, rebuild if the remote branch has updated.
- if test -v TEST_OPENSSL3_BRANCH -a -f $HOME/root/openssl-is-${TEST_OPENSSL3}; then
- latest=`git ls-remote https://github.com/openssl/openssl refs/heads/${TEST_OPENSSL3_BRANCH} | cut -f1`
- : Got branch latest commit ${latest}
- if grep -q ^${latest} $HOME/root/openssl-is-${TEST_OPENSSL3}; then
- : Cached repos already at ${latest}
- else
- : Forcing rebuild
- rm -f $HOME/root/openssl-is-${TEST_OPENSSL3}
- fi
- fi
-
- if ! test -f $HOME/root/openssl-is-${TEST_OPENSSL3}; then
- # Remove any previous install.
- rm -rf $HOME/root/openssl3
-
+ # The cache key covers $TEST_OPENSSL3, $OPENSSL_CONFIG and, for a
+ # branch build, the resolved commit, so an install found here is
+ # current.
+ if ! test -d $HOME/root/openssl3; then
mkdir -p build/openssl
pushd build/openssl
if test -v TEST_OPENSSL3_BRANCH; then
git clone --depth=1 -b $TEST_OPENSSL3_BRANCH -q https://github.com/openssl/openssl openssl-${TEST_OPENSSL3}
+ # Build the commit named in the cache key, not whatever
+ # the branch tip has become since it was resolved.
+ if test -n "${OPENSSL_COMMIT-}"; then
+ git -C openssl-${TEST_OPENSSL3} fetch -q --depth=1 origin ${OPENSSL_COMMIT}
+ git -C openssl-${TEST_OPENSSL3} checkout -q ${OPENSSL_COMMIT}
+ fi
else
curl -L "https://github.com/openssl/openssl/releases/download/openssl-${TEST_OPENSSL3}/openssl-${TEST_OPENSSL3}.tar.gz" |
tar -xzf -
@@ -169,12 +206,6 @@ if test -v TEST_OPENSSL3; then
'-Wl,-rpath=$(LIBRPATH)'
make $MFLAGS
make install_sw
- if test -d .git; then
- : Caching git commit hash:
- git rev-parse HEAD | tee $HOME/root/openssl-is-${TEST_OPENSSL3}
- else
- touch $HOME/root/openssl-is-${TEST_OPENSSL3}
- fi
popd
fi