You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/05/04 17:52:46 UTC
[GitHub] [hbase] mattf-apache opened a new pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
mattf-apache opened a new pull request #1643:
URL: https://github.com/apache/hbase/pull/1643
Clean up `dev-support/create-release` code, fix bugs, remove ambiguities, and clarify usage, to enable:
- Clear statement of the four steps, which are now called `tag`, `publish-dist`, `publish-snapshot`, and `publish-release` (the latter two being mutually exclusive alternatives).
- Ability to do the three tag-dist-release steps with a single command, or do any of the steps singly. (Fixed bugs and unfulfilled dependencies in single-steps.)
- Ability to do a reliable and useful "dry run" of all steps or each step, and chain together the tag step with publish steps in a dry run. (Fixed many dry run bugs.)
- Ability to run any or all steps without bugs in Docker or outside of Docker, on Linux and Mac.
- Cleaned up all `shellcheck` errors in the scripts, and removed ambiguities and redundancies in the many environment variables used.
- Refactoring to make the code more flexible, maintainable, or more easily understood (which will of course be somewhat subjective).
There is more detail in the commit logs attached.
In addition, the changes move the code toward being more general / less HBase-specific, so it can be run on any Apache project (while still accommodating HBase-specific features regarding how sub-projects are named and organized in Jira and release repos). In future I propose to take it further along that path, and move create-release into Yetus (recognizing that this create-release code has been passed between a couple other projects already).
These changes have NO IMPACT on HBase functionality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420502419
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
else
echo "Skipping tag creation for $RELEASE_TAG."
fi
-if should_build "build"; then
- run_silent "Building ${PROJECT}..." "build.log" \
- "$SELF/release-build.sh" build
+if should_build "publish-dist"; then
+ run_silent "Publishing distribution packages (tarballs)" "publish-dist.log" \
+ "$SELF/release-build.sh" publish-dist
else
- echo "Skipping build step."
+ echo "Skipping publish-dist step."
Review comment:
Done, in Release Note in jira (along with the reason why).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r419775446
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -64,24 +65,26 @@ This script runs the release scripts inside a docker image.
Options:
-d [path] required. working directory. output will be written to "output" in here.
- -n dry run mode. Checks and local builds, but does not upload anything.
+ -f "force" -- actually publish this release. Unless you specify '-f', it will
+ default to dry run mode, which checks and does local builds, but does not upload anything.
Review comment:
Highlight this? Add to the release note so RMs notice.
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
else
echo "Skipping tag creation for $RELEASE_TAG."
fi
-if should_build "build"; then
- run_silent "Building ${PROJECT}..." "build.log" \
- "$SELF/release-build.sh" build
+if should_build "publish-dist"; then
+ run_silent "Publishing distribution packages (tarballs)" "publish-dist.log" \
+ "$SELF/release-build.sh" publish-dist
else
- echo "Skipping build step."
+ echo "Skipping publish-dist step."
Review comment:
publish-dist is new step? If so, put that in release note too... as change.
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
Is this what I do when I want to run a release candidate? Release candidate goes into staging repo first.
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -64,24 +65,26 @@ This script runs the release scripts inside a docker image.
Options:
-d [path] required. working directory. output will be written to "output" in here.
- -n dry run mode. Checks and local builds, but does not upload anything.
+ -f "force" -- actually publish this release. Unless you specify '-f', it will
+ default to dry run mode, which checks and does local builds, but does not upload anything.
-t [tag] tag for the hbase-rm docker image to use for building (default: "latest").
-j [path] path to local JDK installation to use building. By default the script will
use openjdk8 installed in the docker image.
- -p [project] project to build; default 'hbase'; alternatively, 'hbase-thirdparty', etc.
- -s [step] runs a single step of the process; valid steps are: tag, build, publish. if
- none specified, runs tag, then build, and then publish.
+ -p [project] project to build, such as 'hbase' or 'hbase-thirdparty'; defaults to $PROJECT env var
+ -s [step] runs a single step of the process; valid steps are: tag|publish-dist|publish-release.
+ If none specified, runs tag, then publish-dist, and then publish-release.
+ 'publish-snapshot' is also an allowed, less used, option.
Review comment:
good
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
Review comment:
I don't know where this comes from. The tag is immediate in my experience. A short wait and just move on would be nice future-to-have.
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
Review comment:
Do you think we need an entrance script or a rename of the one of these scripts as main.sh or some such. As is, operator has to consult the README. I suppose this ok given how specialized this all is.
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
Review comment:
Good doc.
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
For example:
- $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh build
+ $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist
EOF
exit 1
}
set -e
function cleanup {
- echo "Cleaning up temp settings file." >&2
- rm "${tmp_settings}" &> /dev/null || true
# If REPO was set, then leave things be. Otherwise if we defined a repo clean it out.
- if [[ -z "${REPO}" ]] && [[ -n "${tmp_repo}" ]]; then
- echo "Cleaning up temp repo in '${tmp_repo}'. set REPO to reuse downloads." >&2
- rm -rf "${tmp_repo}" &> /dev/null || true
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
fi
}
-if [ $# -eq 0 ]; then
+if [ $# -ne 1 ]; then
exit_with_usage
fi
-if [[ $@ == *"help"* ]]; then
+if [[ "$*" == *"help"* ]]; then
exit_with_usage
fi
-# Read in the ASF password.
-if [[ -z "$ASF_PASSWORD" ]]; then
- echo 'The environment variable ASF_PASSWORD is not set. Enter the password.'
- echo
- stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
-fi
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks)
+perl --version | grep 'This is'
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
+rm -rf "${PROJECT}"
+
+if [[ "$1" == "tag" ]]; then
+ # for 'tag' stage
+ set -o pipefail
+ set -x # detailed logging during action
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD RELEASE_VERSION RELEASE_TAG NEXT_VERSION \
+ GIT_EMAIL GIT_NAME GIT_BRANCH
+ ASF_REPO="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ encoded_username="$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''')")"
+ encoded_password="$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''')")"
+ git clone "https://$encoded_username:$encoded_password@$ASF_REPO" -b "$GIT_BRANCH" "${PROJECT}"
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, all the 'hbase-*'
+ # projects share the same HBASE jira name. To make this work, by convention, the HBASE jira "Fix Version"
+ # field values have the sub-project name pre-pended, as in "hbase-operator-tools-1.0.0".
+ # So, here we prepend the project name to the version, but only for the hbase sub-projects.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" =~ ^hbase- ]]; then
+ jira_fix_version="${PROJECT}-${RELEASE_VERSION}"
+ fi
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
- if [ -z "${!env}" ]; then
- echo "ERROR: $env must be set to run this script"
- exit_with_usage
+ # Create release version
Review comment:
Release or release candidate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420526164
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
For example:
- $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh build
+ $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist
EOF
exit 1
}
set -e
function cleanup {
- echo "Cleaning up temp settings file." >&2
- rm "${tmp_settings}" &> /dev/null || true
# If REPO was set, then leave things be. Otherwise if we defined a repo clean it out.
- if [[ -z "${REPO}" ]] && [[ -n "${tmp_repo}" ]]; then
- echo "Cleaning up temp repo in '${tmp_repo}'. set REPO to reuse downloads." >&2
- rm -rf "${tmp_repo}" &> /dev/null || true
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
fi
}
-if [ $# -eq 0 ]; then
+if [ $# -ne 1 ]; then
exit_with_usage
fi
-if [[ $@ == *"help"* ]]; then
+if [[ "$*" == *"help"* ]]; then
exit_with_usage
fi
-# Read in the ASF password.
-if [[ -z "$ASF_PASSWORD" ]]; then
- echo 'The environment variable ASF_PASSWORD is not set. Enter the password.'
- echo
- stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
-fi
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks)
+perl --version | grep 'This is'
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
+rm -rf "${PROJECT}"
+
+if [[ "$1" == "tag" ]]; then
+ # for 'tag' stage
+ set -o pipefail
+ set -x # detailed logging during action
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD RELEASE_VERSION RELEASE_TAG NEXT_VERSION \
+ GIT_EMAIL GIT_NAME GIT_BRANCH
+ ASF_REPO="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ encoded_username="$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''')")"
+ encoded_password="$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''')")"
+ git clone "https://$encoded_username:$encoded_password@$ASF_REPO" -b "$GIT_BRANCH" "${PROJECT}"
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, all the 'hbase-*'
+ # projects share the same HBASE jira name. To make this work, by convention, the HBASE jira "Fix Version"
+ # field values have the sub-project name pre-pended, as in "hbase-operator-tools-1.0.0".
+ # So, here we prepend the project name to the version, but only for the hbase sub-projects.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" =~ ^hbase- ]]; then
+ jira_fix_version="${PROJECT}-${RELEASE_VERSION}"
+ fi
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
- if [ -z "${!env}" ]; then
- echo "ERROR: $env must be set to run this script"
- exit_with_usage
+ # Create release version
Review comment:
This line of comment comes from the original code from what was formerly `release-tag.sh`. Basically, it sets the maven version in pom.xml for the release or release candidate you are making. It just uses the version string you've specified in do-release.sh, which is usually a release candidate version, but it's up to you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420508959
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
What one should do to run a release candidate, is invoke `do-release-docker.sh` (preferably) or `do-release.sh` (if there's a reason not to use docker). I agree with your comments here and in the next item below, that this script must clearly be documented as NOT the entrance script. And yet, the many env variables that get passed around as inter-script communication do need to be documented somewhere. Let me see if I can move them to `do-release.sh`, and make the comments and the README more useful.
On a completely unrelated note, you mention, as a question,
> Release candidate goes into staging repo first.
The business with staging repo in Nexus is part of the `publish-release` step. I did not change its basic logic, which is now in release-util.sh::maven_deploy():
1. Use mvn to set the version in pom.xml
1. Do `mvn deploy` to Nexus, using `-P apache-release,release` profiles, which use the maven-release-plugin, nexus-staging-maven-plugin, and configs from the HBase top-level pom and root ASF pom, to manage Nexus deployment.
1. Afterward (back to release-build.sh, at the end of `publish-release` step), the code expects to find that the Staging Repo has been closed, and returns the repo ID to the user. Note that for snapshot releases, the Staging Repo intermediate appears not to be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-624448967
@busbey and @ndimiduk You both are about to RM. You could review this as sub-task of RM'ing but may I suggest that you'll have your hands full RM'ing anyways; just take what is here and build on its benefit? It might be a bit hard to review because it does some renaming toward improved clarity and moves some actions around to make more sensible grouping. Just a suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] beettlle commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
beettlle commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r419621936
##########
File path: dev-support/create-release/README.txt
##########
@@ -37,13 +38,14 @@ $ sudo add-apt-repository -y \
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo usermod -a -G docker $USERID
-# LOGOUT and then LOGIN again so $USERID shows as part of docker groupl
+# LOGOUT and then LOGIN again so $USERID shows as part of docker group
Review comment:
Good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-623615747
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420503750
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
Review comment:
Apparently it was in the original code swiped from the Spark project. I think it would be pretty easy to replace it with a little wait loop; I'll do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-625450946
As I said previously, if this is pressing for someone go ahead. I am already on a forked branch in order to fix problems with the release creation tooling as I find them getting towards RC0. As I become sure that the things I am fixing are actually fixed, I'll post them up for reviews. I'm trying to minimize repeated updates or addenda as I figure out that something was only fixed for a part of the RC generation process.
I had wanted to incorporate this change on the branch I'm on (https://github.com/busbey/hbase/tree/HBASE-23339) so that I could ensure they work prior to HBase 3 alpha-1 rather then when I'm trying to maintain a regular cadence of alpha releases. If you are not willing to wait for that for some reason then by all means go ahead. I am likely to ignore them until after alpha-1 RCs are done in that case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420526164
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
For example:
- $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh build
+ $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist
EOF
exit 1
}
set -e
function cleanup {
- echo "Cleaning up temp settings file." >&2
- rm "${tmp_settings}" &> /dev/null || true
# If REPO was set, then leave things be. Otherwise if we defined a repo clean it out.
- if [[ -z "${REPO}" ]] && [[ -n "${tmp_repo}" ]]; then
- echo "Cleaning up temp repo in '${tmp_repo}'. set REPO to reuse downloads." >&2
- rm -rf "${tmp_repo}" &> /dev/null || true
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
fi
}
-if [ $# -eq 0 ]; then
+if [ $# -ne 1 ]; then
exit_with_usage
fi
-if [[ $@ == *"help"* ]]; then
+if [[ "$*" == *"help"* ]]; then
exit_with_usage
fi
-# Read in the ASF password.
-if [[ -z "$ASF_PASSWORD" ]]; then
- echo 'The environment variable ASF_PASSWORD is not set. Enter the password.'
- echo
- stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
-fi
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks)
+perl --version | grep 'This is'
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
+rm -rf "${PROJECT}"
+
+if [[ "$1" == "tag" ]]; then
+ # for 'tag' stage
+ set -o pipefail
+ set -x # detailed logging during action
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD RELEASE_VERSION RELEASE_TAG NEXT_VERSION \
+ GIT_EMAIL GIT_NAME GIT_BRANCH
+ ASF_REPO="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ encoded_username="$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''')")"
+ encoded_password="$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''')")"
+ git clone "https://$encoded_username:$encoded_password@$ASF_REPO" -b "$GIT_BRANCH" "${PROJECT}"
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, all the 'hbase-*'
+ # projects share the same HBASE jira name. To make this work, by convention, the HBASE jira "Fix Version"
+ # field values have the sub-project name pre-pended, as in "hbase-operator-tools-1.0.0".
+ # So, here we prepend the project name to the version, but only for the hbase sub-projects.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" =~ ^hbase- ]]; then
+ jira_fix_version="${PROJECT}-${RELEASE_VERSION}"
+ fi
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
- if [ -z "${!env}" ]; then
- echo "ERROR: $env must be set to run this script"
- exit_with_usage
+ # Create release version
Review comment:
This whole block of code, starting with `if [[ "$1" == "tag" ]]; then` is just cut in from what was formerly `release-tag.sh`. Unfortunately, `diff` interleaved it with random chunks of deleted material from `release-build.sh`, so the flow isn't as obvious as one would wish. The answer to your question is item #3 below:
The flow of `tag` is, briefly:
1. Checkout the code
1. Update Release Notes -- use yetus scripts to scan Jira data and update CHANGES.md and RELEASENOTES.md
1. Set maven version in pom.xml for the release or release candidate you are making. This just uses the version you've specified in do-release.sh, which is usually a release candidate version, but it's up to you.
1. Do `git commit`
1. Do `git tag` to make the tag for the release.
1. Set maven version again, this time to the dev version to be used on this branch _after_ the release.
1. Do `git commit` again.
1. Finally, _if not a dry run,_ do a `git push` to the Apache source repository, thereby "publishing" the tag and the new state of the dev branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-624452897
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 35s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
||| _ Patch Compile Tests _ |
||| _ Other Tests _ |
| | | 2m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1643/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1643 |
| Optional Tests | |
| uname | Linux 5543e1e5eca4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 07077a3950 |
| Max. process+thread count | 41 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1643/2/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-623616069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ master Compile Tests _ |
||| _ Patch Compile Tests _ |
| -0 :warning: | shellcheck | 0m 4s | The patch generated 3 new + 24 unchanged - 96 fixed = 27 total (was 120) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 2m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1643/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1643 |
| Optional Tests | dupname asflicense shellcheck shelldocs |
| uname | Linux 30bcc9274826 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / e37aafcfc2 |
| shellcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1643/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt |
| Max. process+thread count | 49 (vs. ulimit of 12500) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1643/1/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420508959
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
If you don't specify a step, it will do `tag`, `publish-dist` (formerly misnamed as "build"), and `publish-release`, in that order.
The business with staging repo in Nexus is part of the `publish-release` step. I did not change its basic logic, which is now in release-util.sh::maven_deploy():
1. Use mvn to set the version in pom.xml
1. Do `mvn deploy` to Nexus, using `-P apache-release,release` profiles, which use the maven-release-plugin, nexus-staging-maven-plugin, and configs from the HBase top-level pom and root ASF pom, to manage Nexus deployment.
1. Afterward (back to release-build.sh, at the end of `publish-release` step), the code expects to find that the Staging Repo has been closed, and returns the repo ID to the user. Note that for snapshot releases, the Staging Repo intermediate appears not to be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552985
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -64,24 +65,26 @@ This script runs the release scripts inside a docker image.
Options:
-d [path] required. working directory. output will be written to "output" in here.
- -n dry run mode. Checks and local builds, but does not upload anything.
+ -f "force" -- actually publish this release. Unless you specify '-f', it will
+ default to dry run mode, which checks and does local builds, but does not upload anything.
Review comment:
Good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-626195297
Thanks for working through all of this @mattf-apache!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420553720
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
Review comment:
Done. See commit db4bab6f57d6 just added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420526164
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
For example:
- $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh build
+ $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist
EOF
exit 1
}
set -e
function cleanup {
- echo "Cleaning up temp settings file." >&2
- rm "${tmp_settings}" &> /dev/null || true
# If REPO was set, then leave things be. Otherwise if we defined a repo clean it out.
- if [[ -z "${REPO}" ]] && [[ -n "${tmp_repo}" ]]; then
- echo "Cleaning up temp repo in '${tmp_repo}'. set REPO to reuse downloads." >&2
- rm -rf "${tmp_repo}" &> /dev/null || true
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
fi
}
-if [ $# -eq 0 ]; then
+if [ $# -ne 1 ]; then
exit_with_usage
fi
-if [[ $@ == *"help"* ]]; then
+if [[ "$*" == *"help"* ]]; then
exit_with_usage
fi
-# Read in the ASF password.
-if [[ -z "$ASF_PASSWORD" ]]; then
- echo 'The environment variable ASF_PASSWORD is not set. Enter the password.'
- echo
- stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
-fi
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks)
+perl --version | grep 'This is'
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
+rm -rf "${PROJECT}"
+
+if [[ "$1" == "tag" ]]; then
+ # for 'tag' stage
+ set -o pipefail
+ set -x # detailed logging during action
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD RELEASE_VERSION RELEASE_TAG NEXT_VERSION \
+ GIT_EMAIL GIT_NAME GIT_BRANCH
+ ASF_REPO="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ encoded_username="$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''')")"
+ encoded_password="$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''')")"
+ git clone "https://$encoded_username:$encoded_password@$ASF_REPO" -b "$GIT_BRANCH" "${PROJECT}"
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, all the 'hbase-*'
+ # projects share the same HBASE jira name. To make this work, by convention, the HBASE jira "Fix Version"
+ # field values have the sub-project name pre-pended, as in "hbase-operator-tools-1.0.0".
+ # So, here we prepend the project name to the version, but only for the hbase sub-projects.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" =~ ^hbase- ]]; then
+ jira_fix_version="${PROJECT}-${RELEASE_VERSION}"
+ fi
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
- if [ -z "${!env}" ]; then
- echo "ERROR: $env must be set to run this script"
- exit_with_usage
+ # Create release version
Review comment:
This whole block of code, starting with `if [[ "$1" == "tag" ]]; then` is just cut in from what was formerly `release-tag.sh`. Unfortunately, `diff` interleaved it with random chunks of deleted material from `release-build.sh`, so the flow isn't as obvious as one would wish. The answer to your question is item #3 below:
The flow of `tag` is, briefly:
1. Checkout the code
1. Update Release Notes -- use yetus scripts to scan Jira data and update CHANGES.md and RELEASENOTES.md
1. Set maven version in pom.xml for the release or release candidate you are making. This just uses the version you've specified in do-release.sh, which is usually a release candidate version, but it's up to you.
1. Do `git commit`
1. Do `git tag` to make the tag for the release.
1. Set maven version again, this time to the dev version to be used on this branch _after_ the release.
1. Finally, do `git commit` again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552406
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
Review comment:
Sweet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552786
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
k. This sounds like it works as it used to which sort of basically worked.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
busbey commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-623755088
I have been working through trying to use the release scripts to roll 3.0.0-alpha-1. unless it is pressing I would like to incorporate this PR into me doing that as a review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420553720
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
Review comment:
Done. See commit db4bab6f57d6 just added.
It waits in 30-second increments up to 5 minutes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552955
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
Review comment:
No hurry. Nice-to-have. Can come later too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420520956
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
Review comment:
I was letting it go due to how specialized it is, but you're right; there's no reason not to make it nice and clear. Will try to improve this; see comment immediately above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552310
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
+
+Used for 'tag' and 'publish' stages:
+ PROJECT - The project to build. No default.
+ RELEASE_VERSION - Version used in pom files for release (e.g. 2.1.2)
+ Required for 'tag'; defaults for 'publish' to the version in pom at GIT_REF
+ RELEASE_TAG - Name of release tag (e.g. 2.1.2RC0), also used by
+ publish-dist as package version name in dist directory path
+ ASF_USERNAME - Username of ASF committer account
+ ASF_PASSWORD - Password of ASF committer account
+ DRY_RUN - 1:true (default), 0:false. If "1", does almost all the work, but doesn't actually
+ publish anything to upstream source or object repositories. It defaults to "1", so if you want
+ to actually publish you have to set '-f' (force) flag in do-release.sh or do-release-docker.sh.
+
+Used only for 'tag':
+ GIT_NAME - Name to use with git
+ GIT_EMAIL - E-mail address to use with git
+ GIT_BRANCH - Git branch on which to make release. Tag is always placed at HEAD of this branch.
+ NEXT_VERSION - Development version after release (e.g. 2.1.3-SNAPSHOT)
+
+Used only for 'publish':
+ GIT_REF - Release tag or commit to build from (defaults to $RELEASE_TAG; only need to
+ separately define GIT_REF if RELEASE_TAG is not actually present as a tag at publish time)
+ If both RELEASE_TAG and GIT_REF are undefined it will default to HEAD of master.
+ GPG_KEY - GPG key id (usually email addr) used to sign release artifacts
+ GPG_PASSPHRASE - Passphrase for GPG key
+ REPO - Set to full path of a directory to use as maven local repo (dependencies cache)
+ to avoid re-downloading dependencies for each stage. It is automatically set if you
+ request full sequence of stages (tag, publish-dist, publish-release) in do-release.sh.
For example:
- $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh build
+ $ PROJECT="hbase-operator-tools" ASF_USERNAME=NAME ASF_PASSWORD=PASSWORD GPG_PASSPHRASE=PASSWORD GPG_KEY=stack@apache.org ./release-build.sh publish-dist
EOF
exit 1
}
set -e
function cleanup {
- echo "Cleaning up temp settings file." >&2
- rm "${tmp_settings}" &> /dev/null || true
# If REPO was set, then leave things be. Otherwise if we defined a repo clean it out.
- if [[ -z "${REPO}" ]] && [[ -n "${tmp_repo}" ]]; then
- echo "Cleaning up temp repo in '${tmp_repo}'. set REPO to reuse downloads." >&2
- rm -rf "${tmp_repo}" &> /dev/null || true
+ if [[ -z "${REPO}" ]] && [[ -n "${MAVEN_LOCAL_REPO}" ]]; then
+ echo "Cleaning up temp repo in '${MAVEN_LOCAL_REPO}'. Set REPO to reuse downloads." >&2
+ rm -f "${MAVEN_SETTINGS_FILE}" &> /dev/null || true
+ rm -rf "${MAVEN_LOCAL_REPO}" &> /dev/null || true
fi
}
-if [ $# -eq 0 ]; then
+if [ $# -ne 1 ]; then
exit_with_usage
fi
-if [[ $@ == *"help"* ]]; then
+if [[ "$*" == *"help"* ]]; then
exit_with_usage
fi
-# Read in the ASF password.
-if [[ -z "$ASF_PASSWORD" ]]; then
- echo 'The environment variable ASF_PASSWORD is not set. Enter the password.'
- echo
- stty -echo && printf "ASF password: " && read ASF_PASSWORD && printf '\n' && stty echo
-fi
+init_locale
+init_java
+init_mvn
+init_python
+# Print out subset of perl version (used in git hooks)
+perl --version | grep 'This is'
-# Read in the GPG passphrase
-if [[ -z "$GPG_PASSPHRASE" ]]; then
- echo 'The environment variable GPG_PASSPHRASE is not set. Enter the passphrase to'
- echo 'unlock the GPG signing key that will be used to sign the release!'
- echo
- stty -echo && printf "GPG passphrase: " && read GPG_PASSPHRASE && printf '\n' && stty echo
- export GPG_PASSPHRASE
- export GPG_TTY=$(tty)
-fi
+rm -rf "${PROJECT}"
+
+if [[ "$1" == "tag" ]]; then
+ # for 'tag' stage
+ set -o pipefail
+ set -x # detailed logging during action
+ check_get_passwords ASF_PASSWORD
+ check_needed_vars PROJECT ASF_USERNAME ASF_PASSWORD RELEASE_VERSION RELEASE_TAG NEXT_VERSION \
+ GIT_EMAIL GIT_NAME GIT_BRANCH
+ ASF_REPO="gitbox.apache.org/repos/asf/${PROJECT}.git"
+ encoded_username="$(python -c "import urllib; print urllib.quote('''$ASF_USERNAME''')")"
+ encoded_password="$(python -c "import urllib; print urllib.quote('''$ASF_PASSWORD''')")"
+ git clone "https://$encoded_username:$encoded_password@$ASF_REPO" -b "$GIT_BRANCH" "${PROJECT}"
+
+ # 'update_releasenotes' searches the project's Jira for issues where 'Fix Version' matches specified
+ # $jira_fix_version. For most projects this is same as ${RELEASE_VERSION}. However, all the 'hbase-*'
+ # projects share the same HBASE jira name. To make this work, by convention, the HBASE jira "Fix Version"
+ # field values have the sub-project name pre-pended, as in "hbase-operator-tools-1.0.0".
+ # So, here we prepend the project name to the version, but only for the hbase sub-projects.
+ jira_fix_version="${RELEASE_VERSION}"
+ shopt -s nocasematch
+ if [[ "${PROJECT}" =~ ^hbase- ]]; then
+ jira_fix_version="${PROJECT}-${RELEASE_VERSION}"
+ fi
+ shopt -u nocasematch
+ update_releasenotes "$(pwd)/${PROJECT}" "${jira_fix_version}"
+
+ cd "${PROJECT}"
+
+ git config user.name "$GIT_NAME"
+ git config user.email "$GIT_EMAIL"
-for env in ASF_USERNAME GPG_PASSPHRASE GPG_KEY; do
- if [ -z "${!env}" ]; then
- echo "ERROR: $env must be set to run this script"
- exit_with_usage
+ # Create release version
Review comment:
k
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420508959
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
What one should do to run a release candidate, is invoke `do-release-docker.sh` (preferably) or `do-release.sh` (if there's a reason not to use docker). I agree with your comments here and in the next item below, that this script must clearly be documented as NOT the entrance script. And yet, the many env variables that get passed around as inter-script communication do need to be documented somewhere. Let me see if I can move them to `do-release.sh`, and make the comments and the README more useful.
On a completely unrelated note, you mention, as a question,
> Release candidate goes into staging repo first.
That's correct, for non-snapshot releases/release candidates. The business with staging repo in Nexus is part of the `publish-release` step. I did not change its basic logic, which is now in release-util.sh::maven_deploy():
1. Use mvn to set the version in pom.xml
1. Do `mvn deploy` to Nexus, using `-P apache-release,release` profiles, which use the maven-release-plugin, nexus-staging-maven-plugin, and configs from the HBase top-level pom and root ASF pom, to manage Nexus deployment.
1. Afterward (back to release-build.sh, at the end of `publish-release` step), the code expects to find that the Staging Repo has been closed (made immutable), and returns the repo ID to the user. Note that for snapshot releases, the Staging Repo intermediate appears not to be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420554402
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
+ construct vote email from template
+
+All other inputs are environment variables. Please use do-release-docker.sh or
+do-release.sh to set up the needed environment variables. This script, release-build.sh,
+is not intended to be called stand-alone, and such use is untested. The env variables used are:
Review comment:
Getting late, will tackle this in the morning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420508959
##########
File path: dev-support/create-release/release-build.sh
##########
@@ -20,114 +20,180 @@
trap cleanup EXIT
# Source in utils.
-SELF=$(cd $(dirname $0) && pwd)
+SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=SCRIPTDIR/release-util.sh
. "$SELF/release-util.sh"
# Print usage and exit.
function exit_with_usage {
- cat << EOF
-Usage: release-build.sh <build|publish-snapshot|publish-release>
-Creates build deliverables from a tag/commit.
-Arguments:
- build Create binary packages and commit to dist.apache.org/repos/dist/dev/hbase/
- publish-snapshot Publish snapshot release to Apache snapshots
- publish-release Publish a release to Apache release repo
-
-All other inputs are environment variables:
- GIT_REF - Release tag or commit to build from
- PACKAGE_VERSION - Release identifier in top level package directory (e.g. 2.1.2RC1)
- VERSION - (optional) Version of project being built (e.g. 2.1.2)
- ASF_USERNAME - Username of ASF committer account
- ASF_PASSWORD - Password of ASF committer account
- GPG_KEY - GPG key used to sign release artifacts
- GPG_PASSPHRASE - Passphrase for GPG key
- PROJECT - The project to build. No default.
-
-Set REPO environment to full path to repo to use
-to avoid re-downloading dependencies on each run.
+ cat <<'EOF'
+Usage: release-build.sh <tag|publish-dist|publish-snapshot|publish-release>
+Creates release deliverables from a tag or commit.
+Argument: one of 'tag', 'publish-dist', 'publish-snapshot', or 'publish-release'
+ tag Prepares for release on specified git branch: Set release version,
+ update CHANGES and RELEASENOTES, create release tag,
+ increment version for ongoing dev, and publish to Apache git repo.
+ publish-dist Build and publish distribution packages (tarballs) to Apache dist repo
+ publish-snapshot Build and publish maven artifacts snapshot release to Apache snapshots repo
+ publish-release Build and publish maven artifacts release to Apache release repo, and
Review comment:
What one should do to run a release candidate, is invoke `do-release-docker.sh` (preferably) or `do-release.sh` (if there's a reason not to use docker). I agree with your comments here and in the next item below, that this script must clearly be documented as NOT the entrance script. And yet, the many env variables that get passed around as inter-script communication do need to be documented somewhere. Let me see if I can move them to `do-release.sh`, and make the comments and the README more useful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-624452673
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420502004
##########
File path: dev-support/create-release/do-release-docker.sh
##########
@@ -64,24 +65,26 @@ This script runs the release scripts inside a docker image.
Options:
-d [path] required. working directory. output will be written to "output" in here.
- -n dry run mode. Checks and local builds, but does not upload anything.
+ -f "force" -- actually publish this release. Unless you specify '-f', it will
+ default to dry run mode, which checks and does local builds, but does not upload anything.
Review comment:
Great idea, added a Release Note in the jira.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] mattf-apache commented on pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
mattf-apache commented on pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#issuecomment-624188067
@saintstack , thank you for the thorough review, I will respond to all comments momentarily.
@busbey , I'd be delighted to have you use it. This can be done two ways:
- We can commit this patch as is, and fix any issues as new bugs. This has the benefit that the improved tools will be included in the 3.0.0-alpha-1 release.
- Or you can take advantage of the fact that these tools don't use the workspace they are called from. Instead, they always create a clean workspace directory, and into that they pull a fresh "official" copy of the code from gitbox.apache.org, at the branch being built. Since the invoking workspace won't be used in the build, you can just apply this patch to a private workspace, or clone https://github.com/mattf-apache/hbase/ and checkout branch `release-tool-enhancements`, and invoke the tools from there. Then we can keep this PR open, and only commit it after your test.
In either case, I'd be happy to respond promptly to any problems you find.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1643: HBASE-24318 Create-release scripts fixes and enhancements
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1643:
URL: https://github.com/apache/hbase/pull/1643#discussion_r420552836
##########
File path: dev-support/create-release/do-release.sh
##########
@@ -66,26 +82,34 @@ function should_build {
fi
}
-if should_build "tag" && [ $SKIP_TAG = 0 ]; then
+if should_build "tag" && [ "$SKIP_TAG" = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
- "$SELF/release-tag.sh"
- echo "It may take some time for the tag to be synchronized to github."
- echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
- read
+ "$SELF/release-build.sh" tag
+ if is_dry_run; then
+ export TAG_SAME_DRY_RUN="true";
+ else
+ echo "It may take some time for the tag to be synchronized to github."
+ echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
+ read -r
+ fi
else
echo "Skipping tag creation for $RELEASE_TAG."
fi
-if should_build "build"; then
- run_silent "Building ${PROJECT}..." "build.log" \
- "$SELF/release-build.sh" build
+if should_build "publish-dist"; then
+ run_silent "Publishing distribution packages (tarballs)" "publish-dist.log" \
+ "$SELF/release-build.sh" publish-dist
else
- echo "Skipping build step."
+ echo "Skipping publish-dist step."
Review comment:
Good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org