You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ot...@apache.org on 2018/10/24 15:04:26 UTC
[37/51] [abbrv] metron git commit: METRON-1769 Script creation of a
release candidate (justinleet) closes apache/metron#1188
METRON-1769 Script creation of a release candidate (justinleet) closes apache/metron#1188
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f153375f
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f153375f
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f153375f
Branch: refs/heads/feature/METRON-1090-stellar-assignment
Commit: f153375fad65e75a10ce490b4a5f29f7e82f70dd
Parents: 3d923cd
Author: justinleet <ju...@gmail.com>
Authored: Wed Oct 10 15:41:22 2018 -0400
Committer: leet <le...@apache.org>
Committed: Wed Oct 10 15:41:22 2018 -0400
----------------------------------------------------------------------
dev-utilities/release-utils/README.md | 129 +++++++
dev-utilities/release-utils/metron-rc-check | 6 +-
.../release-utils/prepare-release-candidate | 337 +++++++++++++++++++
3 files changed, 469 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/f153375f/dev-utilities/release-utils/README.md
----------------------------------------------------------------------
diff --git a/dev-utilities/release-utils/README.md b/dev-utilities/release-utils/README.md
new file mode 100644
index 0000000..d1fd816
--- /dev/null
+++ b/dev-utilities/release-utils/README.md
@@ -0,0 +1,129 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Release Tools
+
+This project contains tools to assist Apache Metron project committers.
+
+## Prepare Release Candidate
+
+This script automates the process of creating a release candidate from `apache/metron` or `apache/metron-bro-plugin-kafka`. The script will prompt for various information necessary. Ensure your signing key is setup per [Release Signing](https://www.apache.org/dev/release-signing.html) and [Apache GnuPGP Instructions](https://www.apache.org/dev/openpgp.html#gnupg)
+
+When prompted the `[value in brackets]` is used by default. To accept the default, simply press `enter`. If you would like to change the default, type it in and hit `enter` when done.
+
+In the following example, enter the appropriate information
+
+1. Execute the script.
+
+ The first time the script is run, you will be prompted for additional information including your Apache username and Apache email. These values are persisted in `~/.metron-prepare-release-candidate`. Subsequent executions of the script will retrieve these values, rather than prompting you again for them.
+
+ ```
+ $ ./prepare-release-candidate
+ your apache userid []: leet
+ your apache email [leet@apache.org]:
+ ```
+
+1. Select a repository we're creating an RC for.
+
+ ```
+ [1] metron
+ [2] metron-bro-plugin-kafka
+ which repo? [1]: 1
+ ```
+
+1. Enter the current version number. This will be the base for the CHANGES file
+
+ ```
+ current version: 0.6.0
+ ```
+
+1. Enter the version being built.
+
+ ```
+ version being built: 0.6.1
+ ```
+
+1. Enter the current RC number
+
+ ```
+ release candidate number: 1
+ ```
+
+1. Enter the branch we're releasing from. In most cases, this will be master, but for maintenance releases it can be another branch.
+
+ ```
+ base revision branch or hash for release candidate [master]:
+ ```
+
+1. Enter the signing key id.
+
+ ```
+ signing key id in 8-byte format (e.g. BADDCAFEDEADBEEF):
+ ```
+
+1. Enter if this is a practice run. In a practice run, nothing is pushed to SVN, but everything is setup and built otherwise.
+
+ ```
+ do a live run (push to remote repositories?) [y/n]
+ ```
+
+1. Wait for all repos to be checked out to complete. There will be some additional work done, e.g. along with branch and tag creation. In a live run, you may be prompted for Git credentials to push a branch.
+
+ ```
+ Checking out repo: https://dist.apache.org/repos/dist/dev/metron
+ Checking out repo: dev
+ Checking out repo: https://dist.apache.org/repos/dist/release/metron
+ Checking out repo: release
+ Checking out git repo: https://git-wip-us.apache.org/repos/asf/metron.git
+ Cloning into '/Users/justinleet/tmp/metron-0.6.1/metron'...
+ remote: Counting objects: 46146, done.
+ remote: Compressing objects: 100% (15568/15568), done.
+ remote: Total 46146 (delta 21513), reused 43696 (delta 19489)
+ Receiving objects: 100% (46146/46146), 56.00 MiB | 1.04 MiB/s, done.
+ Resolving deltas: 100% (21513/21513), done.
+ Creating branch: Metron_0.6.1
+ Using git rev: master
+ Already on 'master'
+ Your branch is up to date with 'origin/master'.
+ Switched to a new branch 'Metron_0.6.1'
+ This is a practice run. Not running <git push --set-upstream origin Metron_0.6.1>
+ Creating tentative git tag <0.6.1-rc1>. Do not push this tag until RC is ready for community review.
+ Already on 'Metron_0.6.1'
+ Creating the RC tarball for tag apache-metron-0.6.1-rc1
+ Creating the SHA hash files
+ ```
+
+1. Provide the passphrase to `gpg` to sign the artifacts.
+
+ ```
+ Signing the release tarball
+ Copying release artifacts
+ ```
+
+1. Shortly afterwards the RC will be finalized. In a practice run, this will not be pushed back to SVN.
+ ```
+ Creating CHANGES file
+ Extracting LICENSE, NOTICE, and KEYS from tarball
+ x LICENSE
+ x NOTICE
+ This is a practice run. Not running the following commands:
+ <svn add 0.6.1-RC1>
+ <svn commit -m "Adding artifacts for metron 0.6.1-RC1">
+ ```
+
+At this point, all RC artifacts have been created. In a live run, these will have been pushed to the appropriate repositories and are ready for community review.
http://git-wip-us.apache.org/repos/asf/metron/blob/f153375f/dev-utilities/release-utils/metron-rc-check
----------------------------------------------------------------------
diff --git a/dev-utilities/release-utils/metron-rc-check b/dev-utilities/release-utils/metron-rc-check
index e3cc39d..143ba85 100755
--- a/dev-utilities/release-utils/metron-rc-check
+++ b/dev-utilities/release-utils/metron-rc-check
@@ -181,7 +181,7 @@ fi
if [ -n "$BRO" ]; then
- METRON_KAFKA_BRO_ASSEMBLY="$METRON_RC_DIST/apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz"
+ METRON_KAFKA_BRO_ASSEMBLY="$METRON_RC_DIST/apache-metron-bro-plugin-kafka_$BRO_VERSION-$RC.tar.gz"
METRON_KAFKA_BRO_ASSEMBLY_ASC="$METRON_KAFKA_BRO_ASSEMBLY.asc"
echo "Downloading $METRON_KAFKA_BRO_ASSEMBLY"
@@ -214,12 +214,12 @@ fi
if [ -n "$BRO" ]; then
echo "Verifying Bro Kafka Plugin Assembly"
- if ! gpg --verify ./"apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz.asc" "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then
+ if ! gpg --verify ./"apache-metron-bro-plugin-kafka_$BRO_VERSION-$RC.tar.gz.asc" "apache-metron-bro-plugin-kafka_$BRO_VERSION-$RC.tar.gz" ; then
echo "[ERROR] failed to verify Bro Kafka Plugin Assembly"
exit 1
fi
- if ! tar -xzf "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then
+ if ! tar -xzf "apache-metron-bro-plugin-kafka_$BRO_VERSION-$RC.tar.gz" ; then
echo "[ERROR] failed to unpack Bro Kafka Plugin Assembly"
exit 1
fi
http://git-wip-us.apache.org/repos/asf/metron/blob/f153375f/dev-utilities/release-utils/prepare-release-candidate
----------------------------------------------------------------------
diff --git a/dev-utilities/release-utils/prepare-release-candidate b/dev-utilities/release-utils/prepare-release-candidate
new file mode 100755
index 0000000..bedee9b
--- /dev/null
+++ b/dev-utilities/release-utils/prepare-release-candidate
@@ -0,0 +1,337 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# Creates and sets up the actual release artifacts into dev
+# See https://cwiki.apache.org/confluence/display/METRON/Release+Process
+
+
+set -eo pipefail
+
+# define constants
+# Git repos
+# To add a new submodule, add the repo name, upstream Git repo, and update the getrepo() selection function.
+# if versioning of the submodule isn't x.y.z format, retrieval of the git tag must also be adjusted.
+METRON_REPO_NAME="metron"
+BRO_PLUGIN_REPO_NAME="metron-bro-plugin-kafka"
+METRON_UPSTREAM="https://git-wip-us.apache.org/repos/asf/${METRON_REPO_NAME}.git"
+BRO_PLUGIN_UPSTREAM="https://git-wip-us.apache.org/repos/asf/${BRO_PLUGIN_REPO_NAME}.git"
+
+DEV_REPO="https://dist.apache.org/repos/dist/dev/metron"
+RELEASE_REPO=" https://dist.apache.org/repos/dist/release/metron"
+PLUGIN_GIT_REPO="https://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka.git"
+
+RC_PREFIX=rc
+TAG_POSTFIX="-release"
+
+CONFIG_FILE=~/.metron-prepare-release-candidate
+# does a config file already exist?
+if [ -f $CONFIG_FILE ]; then
+ . $CONFIG_FILE
+ echo " ...using settings from $CONFIG_FILE"
+fi
+
+# apache id of committer (you)
+if [ -z "${APACHE_NAME}" ]; then
+ read -p " your apache userid [${APACHE_NAME}]: " INPUT
+ [ -n "$INPUT" ] && APACHE_NAME=$INPUT
+
+ # write setting to config file
+ echo "APACHE_NAME=$APACHE_NAME" >> $CONFIG_FILE
+fi
+
+# apache email addr of committer (you)
+if [ -z "${APACHE_EMAIL}" ]; then
+ APACHE_EMAIL=${APACHE_NAME}@apache.org
+ read -p " your apache email [${APACHE_EMAIL}]: " INPUT
+ [ -n "$INPUT" ] && APACHE_EMAIL=$INPUT
+
+ # write setting to config file, so it is not needed next time
+ echo "APACHE_EMAIL=$APACHE_EMAIL" >> $CONFIG_FILE
+fi
+
+# which repo? metron or metron-bro-plugin-kafka
+getrepo() {
+ echo " [1] ${METRON_REPO_NAME}"
+ echo " [2] ${BRO_PLUGIN_REPO_NAME}"
+ read -p " which repo? [1]: " INPUT
+ case "${INPUT}" in
+ [Bb][Rr][Oo]|[Mm][Ee][Tt][Rr][Oo][Nn]-[Bb][Rr][Oo]-[Pp][Ll][Uu][Gg][Ii][Nn]-[Kk][Aa][Ff][Kk][Aa]|*metron-bro-plugin-kafka\.git|2)
+ INPUT="${BRO_PLUGIN_UPSTREAM}" ;;
+ [Mm][Ee][Tt][Rr][Oo][Nn]|*metron\.git|1|'')
+ INPUT="${METRON_UPSTREAM}" ;;
+ *)
+ echo "Invalid repo, provided \"${INPUT}\". Please choose between ${METRON_REPO_NAME} or ${BRO_PLUGIN_REPO_NAME}"
+ return 1
+ ;;
+ esac
+ [ -n "$INPUT" ] && UPSTREAM=$INPUT
+ return 0
+}
+until getrepo; do :; done
+
+CHOSEN_REPO=$(basename ${UPSTREAM%%.git})
+# Need the capitalized version of the repos some naming
+CAPITAL_REPO="$(tr '[:lower:]' '[:upper:]' <<< ${CHOSEN_REPO:0:1})${CHOSEN_REPO:1}"
+
+getcurrentversion() {
+ # currently released version. Used for CHANGES file
+ read -p " current version: " CURRENT_VERSION
+ if ! [[ "${CURRENT_VERSION}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+ printf " Please enter a valid x.y.z version\n"
+ return 1
+ fi
+ return 0
+}
+until getcurrentversion; do : ; done
+
+# version that we're building an RC for
+getversion() {
+ read -p " version being built: " VERSION
+ if ! [[ "${VERSION}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+ printf " Please enter a valid x.y.z version\n"
+ return 1
+ fi
+ return 0
+ }
+until getversion; do : ; done
+
+# RC number we're building
+getrcnum() {
+read -p " release candidate number: " RC_NUM
+ if [[ "${RC_NUM}" =~ ^[0-9]+$ ]]; then
+ RC="${RC_PREFIX}${RC_NUM}"
+ return 0
+ else
+ printf " Please enter an integer\n"
+ return 1
+ fi
+}
+until getrcnum; do : ; done
+
+# define default values
+TMPDIR="$HOME/tmp"
+WORKDIR="$TMPDIR/${CHOSEN_REPO}-${VERSION}"
+
+# warn the user if the working directory exists
+if [ -d "$WORKDIR" ]; then
+ read -p " directory exists [$WORKDIR]. overwrite existing directory? [y/n] " -r
+ echo
+ if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+ exit 1
+ fi
+fi
+
+# Clear out the existing work directory
+rm -rf "$WORKDIR"
+mkdir "$WORKDIR"
+
+getbaserev() {
+ read -p " base revision branch or hash for release candidate [master]: " GIT_REF
+ GIT_REF=${GIT_REF:-master}
+
+ # check to see if we were given branch
+ git rev-parse -q --verify ${GIT_REF} >> /dev/null
+ if [[ $? -ne 0 ]]; then
+ # check to see if we were given hash
+ git cat-file -e ${GIT_REF}
+ if [[ $? -ne 0 ]]; then
+ "Unable to find git revision"
+ return 1
+ fi
+ fi
+ return 0
+}
+until getbaserev; do : ; done
+
+# Signing key
+getkey() {
+ read -s -p " signing key id in 8-byte format (e.g. BADDCAFEDEADBEEF): " SIGNING_KEY
+ printf "\n"
+ if ! [[ "${SIGNING_KEY}" =~ ^[A-F0-9]{16}$ ]]; then
+ printf " Please enter a valid signing key\n"
+ return 1
+ fi
+ return 0
+}
+
+until getkey; do : ; done
+
+# Determine if this is a practice run or not.
+getpractice() {
+ read -p " do a live run (push to remote repositories?) [y/n] " INPUT
+ case "${INPUT}" in
+ y)
+ PRACTICE_RUN=false
+ return 0 ;;
+ n)
+ PRACTICE_RUN=true
+ return 0 ;;
+ *)
+ printf "Please enter 'y' or 'n'\n"
+ return 1
+ ;;
+ esac
+}
+until getpractice; do : ; done
+
+## Prepare everything for building the release artifacts
+
+# Fetch the SVN repos. Always needed regardless of what's being released.
+fetch_svn_repo () {
+ local SVN_REPO=$1
+ local SVN_DIR=$2
+ mkdir "$SVN_DIR"
+ cd "$SVN_DIR"
+ printf "Checking out repo: %s\n" "$SVN_REPO" "$(basename $SVN_DIR)"
+ svn co -q $SVN_REPO
+ cd "$(dirname $WORKDIR)"
+}
+
+fetch_svn_repo "$DEV_REPO" "$WORKDIR/dev"
+fetch_svn_repo "$RELEASE_REPO" "$WORKDIR/release"
+
+# Fetch the appropriate Git repo. Only need what we're releasing
+GIT_DIR="$WORKDIR/${CHOSEN_REPO}"
+printf "Checking out git repo: %s\n" "$UPSTREAM"
+git clone $UPSTREAM "${GIT_DIR}"
+cd "${GIT_DIR}"
+git fetch --tags
+
+# Create the release branch in the Git repo
+printf "Creating branch: %s_%s\n" "${CAPITAL_REPO}" "$VERSION"
+printf "Using git rev: %s\n" ${GIT_REF}
+cd "$GIT_DIR"
+git checkout ${GIT_REF}
+git checkout -b "${CAPITAL_REPO}_${VERSION}"
+
+if [ "${PRACTICE_RUN}" = true ]; then
+ printf "This is a practice run. Not running <git push --set-upstream origin %s_%s>\n" "${CAPITAL_REPO}" "$VERSION"
+else
+ printf "Pushing branch %s_%s\n" "${CAPITAL_REPO}" "$VERSION"
+ git push --set-upstream origin "${BRANCH_PREFIX}_${BRANCH_VERSION}"
+fi
+
+# Create directory for release artifacts
+if [ "${CHOSEN_REPO}" = "${METRON_REPO_NAME}" ]; then
+ ART_DIR="$WORKDIR/dev/metron/${VERSION}-RC${RC_NUM}"
+else
+ # We're using a sub module, so put it in it's own directory.
+ ART_DIR="$WORKDIR/dev/metron/${CHOSEN_REPO}/${VERSION}-RC${RC_NUM}"
+fi
+mkdir -p "$ART_DIR"
+
+# Setup various parameters we need for the release artifacts
+if [ "${CHOSEN_REPO}" = "${METRON_REPO_NAME}" ]; then
+ CORE_PREFIX="apache-metron-"
+ ARTIFACT_PREFIX="${CORE_PREFIX}"
+ TAG_VERSION="${CURRENT_VERSION}"
+ TAG="${CORE_PREFIX}${TAG_VERSION}${TAG_POSTFIX}"
+elif [ "${CHOSEN_REPO}" = "${BRO_PLUGIN_REPO_NAME}" ]; then
+ PLUGIN_PREFIX="apache-metron-bro-plugin-kafka_"
+ ARTIFACT_PREFIX="${PLUGIN_PREFIX}"
+ TAG_VERSION="${CURRENT_VERSION}"
+ TAG="${PLUGIN_PREFIX}${TAG_VERSION}${TAG_POSTFIX}"
+ # Handle special tag case from prior release
+ if [ "${TAG_VERSION}" = "0.1" ]; then
+ TAG="0.1"
+ fi
+else
+ # If we ever add new modules, add them as needed.
+ printf "Unrecognized module: %s\n" "${CHOSEN_REPO}"
+ exit 1
+fi
+ARTIFACT="${ARTIFACT_PREFIX}${VERSION}-${RC}"
+
+## Do the work of actually creating the release artifacts
+printf "Creating tentative git tag <%s%s-%s>. Do not push this tag until RC is ready for community review.\n" "${TAG_PREFIX}" "$VERSION" "$RC"
+cd "$GIT_DIR"
+git checkout "${CAPITAL_REPO}_${VERSION}"
+# The branch only exists if this is not a practice run
+if [ "${PRACTICE_RUN}" = false ]; then
+ printf "Pulling latest state of branch\n"
+ git pull
+fi
+git tag "${ARTIFACT}"
+
+# Create the rc tarball from the tag
+printf "Creating the RC tarball for tag %s\n" "$ARTIFACT"
+git archive "--prefix=${ARTIFACT}/" "${ARTIFACT}" | gzip > "${ARTIFACT}.tar.gz"
+
+# Create signing hash files
+printf "Creating the SHA hash files\n"
+gpg --print-md SHA512 ${ARTIFACT}.tar.gz > ${ARTIFACT}.tar.gz.sha512
+gpg --print-md SHA256 ${ARTIFACT}.tar.gz > ${ARTIFACT}.tar.gz.sha256
+
+# Sign the release tarball
+printf "Signing the release tarball\n"
+gpg -u ${SIGNING_KEY} --armor --output ${ARTIFACT}.tar.gz.asc --detach-sig ${ARTIFACT}.tar.gz
+if [[ $? -ne 0 ]]; then
+ # gpg will print out an error on its own
+ exit 1
+fi
+
+# Setup the release artifacts
+printf "Copying release artifacts\n"
+mv "${GIT_DIR}/${ARTIFACT}.tar.gz" "$ART_DIR"
+mv "${ARTIFACT}.tar.gz.sha512" "$ART_DIR"
+mv "${ARTIFACT}.tar.gz.sha256" "$ART_DIR"
+mv "${ARTIFACT}.tar.gz.asc" "$ART_DIR"
+
+# Create the CHANGES file
+# Do this by getting all commits in current branch that aren't in current release. Filter out any merges by making sure lines start with blankspace followed by "METRON"
+# i.e. make sure the lines starts with a ticket number to avoid merge commits into feature branches
+printf "Creating CHANGES file\n"
+git log "${CAPITAL_REPO}_${VERSION}" "^tags/${TAG}" --no-merges | grep -E "^[[:blank:]]+METRON" | sed 's/\[//g' | sed 's/\]//g' | grep -v "http" > "${ART_DIR}/CHANGES"
+if [[ $? -ne 0 ]]; then
+ "Error creating CHANGES file"
+ exit 1
+fi
+
+printf "Extracting LICENSE, NOTICE, and KEYS from tarball\n" # Only pull from core
+cd ${ART_DIR}
+
+if [ "${CHOSEN_REPO}" = "${BRO_PLUGIN_REPO_NAME}" ]; then
+ # Bro's convention for the LICENSE is different, so the file is called COPYING
+ tar --strip-components=1 -zxvf "${ARTIFACT}.tar.gz" "${ARTIFACT}/COPYING"
+else
+ tar --strip-components=1 -zxvf "${ARTIFACT}.tar.gz" "${ARTIFACT}/LICENSE"
+fi
+
+# TODO figure out what to do for bro repo here. The KEYS file only needs to live in the /dist root, rather than in each sub repo.
+# Should we have a separate process for adding to the KEYS file without doing a Metron release?
+#tar --strip-components=1 -zxvf "${ARTIFACT}.tar.gz" "${ARTIFACT}/KEYS"
+tar --strip-components=1 -zxvf "${ARTIFACT}.tar.gz" "${ARTIFACT}/NOTICE"
+
+# Add the directory and commit to subversion
+COMMIT_DIR=$(basename ${ART_DIR})
+if [ "${PRACTICE_RUN}" = true ]; then
+ printf "This is a practice run. Not running the following commands:\n"
+ printf "<svn add %s>\n" ${COMMIT_DIR}
+ printf "<svn commit -m \"Adding artifacts for %s %s\">\n" "${CHOSEN_REPO}" "${COMMIT_DIR}"
+else
+ printf "Adding artifacts for Metron ${VERSION}-RC${RC_NUM} to dev SVN\n"
+ # Metron goes in the root of the dir, submodules go in folder
+ if [ "${CHOSEN_REPO}" = "${METRON_REPO_NAME}" ]; then
+ cd "$WORKDIR/dev/metron/"
+ else
+ cd "$WORKDIR/dev/metron/${CHOSEN_REPO}"
+ fi
+ svn add ${COMMIT_DIR}
+ svn commit -m "Adding artifacts for ${CHOSEN_REPO} ${COMMIT_DIR}"
+fi
+