(daffodil-infrastructure) branch main updated: Modify check-release script to work with SBT and VS Code releases

[sl...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

slawrence pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/daffodil-infrastructure.git


The following commit(s) were added to refs/heads/main by this push:
     new 0ccfc7e  Modify check-release script to work with SBT and VS Code releases
0ccfc7e is described below

commit 0ccfc7ef770463b9d73e75c4361c0a4bb84a695f
Author: Steve Lawrence <sl...@apache.org>
AuthorDate: Fri Aug 29 13:23:04 2025 -0400

    Modify check-release script to work with SBT and VS Code releases
    
    The dist directory for SBT and VS Code releases includes an extra
    'daffodil-sbt/' or 'daffodil-vscode/' directory, compared to normal
    daffodil releases. And the value provided to --cut-dir does not consider
    these extra directories, so downloaded directory tree structure doesn't
    actaully match the structure created by the build-release container,
    causing the reproducible diff to fail.
    
    To avoid needing to detect and specify different cut dir values based on
    the release type, this moves the wget logic to a new 'download_dir'
    function that calculates the value of cut-dirs so that it only keeps the
    last directory.
    
    So for example, all dist artifacts are downloaded to a
    directory matching the version (e.g. 1.0.0-rc1/) and all maven artifacts
    are downloaded to a directory matching the maven profile (e.g.
    orgapachedaffodil-1234/). Note that the build-container does not include
    the maven profile directory, so after it is downloaded we move the
    contents out of that directory.
    
    This matches the structure that the build-release container creates
    allowing reproducible diffs to work.
    
    DAFFODIL-2971
---
 containers/check-release/src/check-release.sh | 37 ++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/containers/check-release/src/check-release.sh b/containers/check-release/src/check-release.sh
index 4118c52..fd67462 100755
--- a/containers/check-release/src/check-release.sh
+++ b/containers/check-release/src/check-release.sh
@@ -22,6 +22,31 @@ then
 	exit 1
 fi
 
+download_dir() {
+	# force a trailing slash by removing a slash it if exists and then adding
+	# it back. This is required for maven URLs to download the right thing--it
+	# doesn't matter either way for dist urls
+	URL="${1%/}/"
+	# non-greedily delete the schema and domain part of the URL, giving just
+	# the path part
+	URL_PATH="${URL#*://*/}"
+	# we want to use --cut-dirs to ignore all directories but the final target
+	# directory. This number is different depending on the URL, so we extract
+	# and count the nubmer of slash-separated fields of the url path and
+	# subtract 2 (one because we want to keep the last field, and one because
+	# there is an extra field because the path ends in a slash)
+	CUT_DIRS=$(echo "$URL_PATH" | awk -F/ '{print NF - 2}')
+	wget \
+		--recursive \
+		--level=inf \
+		-e robots=off \
+		--no-parent \
+		--no-host-directories \
+		--reject=index.html,robots.txt \
+		--cut-dirs=$CUT_DIRS \
+		"$URL"
+}
+
 # URL of release candidate directory in dev/dist/, e.g. https://dist.apache.org/repos/dist/dev/daffodil/1.0.0-rc1
 DIST_URL=$1
 
@@ -45,7 +70,6 @@ require_command sha1sum
 require_command sha512sum
 require_command wget
 
-WGET="wget --recursive --level=inf -e robots=off --no-parent --no-host-directories --reject=index.html,robots.txt"
 
 RELEASE_DIR=release-download
 DIST_DIR=$RELEASE_DIR/asf-dist
@@ -56,16 +80,21 @@ printf "\n==== Downloading Release Files ====\n"
 # download dist/dev/ files
 mkdir -p $DIST_DIR
 pushd $DIST_DIR &>/dev/null
-$WGET --cut-dirs=4 $DIST_URL/
+download_dir $DIST_URL
 popd &>/dev/null
 
-# download maven repository, delete nexus generated files
+# download maven repository, delete nexus generated files, and remove the
+# orgapachedaffodil-1234 dir since the build-release container does not have
+# this directory
 if [ -n "$MAVEN_URL" ]
 then
 	mkdir -p $MAVEN_DIR
 	pushd $MAVEN_DIR &>/dev/null
-	$WGET --cut-dirs=3 $MAVEN_URL/
+	download_dir $MAVEN_URL
 	find . -type f \( -name 'archetype-catalog.xml' -o -name 'maven-metadata.xml*' \) -delete
+	REPO_DIR=(*/)
+	mv $REPO_DIR/* .
+	rmdir $REPO_DIR
 	popd &>/dev/null
 fi