You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2021/05/24 18:02:47 UTC
[arrow-rs] branch master updated: Document and automate new release
process (#299)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new 5295e25 Document and automate new release process (#299)
5295e25 is described below
commit 5295e253c46f217a7b75dd8d7f3d751cfc34a460
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Mon May 24 14:02:34 2021 -0400
Document and automate new release process (#299)
* Add Release README and Scripts to create and release tarballs
* Suggestions from Andy Grove
---
arrow/README.md | 31 +--
dev/release/README.md | 236 ++++++++++++++++++++++
dev/release/check-rat-report.py | 59 ++++++
dev/release/create-tarball.sh | 123 +++++++++++
dev/release/release-tarball.sh | 72 +++++++
change_log.sh => dev/release/update_change_log.sh | 15 +-
6 files changed, 506 insertions(+), 30 deletions(-)
diff --git a/arrow/README.md b/arrow/README.md
index 674c3fc..7c54da0 100644
--- a/arrow/README.md
+++ b/arrow/README.md
@@ -174,33 +174,6 @@ specific JIRA issues and reference them in these code comments. For example:
// This is not sound because .... see https://issues.apache.org/jira/browse/ARROW-nnnnn
```
-# Publishing to crates.io
+# Releases and publishing to crates.io
-An Arrow committer can publish this crate after an official project release has
-been made to crates.io using the following instructions.
-
-Follow [these
-instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to
-create an account and login to crates.io before asking to be added as an owner
-of the [arrow crate](https://crates.io/crates/arrow).
-
-Checkout the tag for the version to be released. For example:
-
-```bash
-git checkout apache-arrow-0.11.0
-```
-
-If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it
-should) then the crate can be published with the following command:
-
-```bash
-cargo publish
-```
-
-If the Cargo.toml does not have the correct version then it will be necessary
-to modify it manually. Since there is now a modified file locally that is not
-committed to GitHub it will be necessary to use the following command.
-
-```bash
-cargo publish --allow-dirty
-```
+Please see the [release](../dev/release/README.md) for details on how to create arrow releases
diff --git a/dev/release/README.md b/dev/release/README.md
new file mode 100644
index 0000000..9581a63
--- /dev/null
+++ b/dev/release/README.md
@@ -0,0 +1,236 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Release Process
+
+## Branching
+
+We would maintain two branches: `active_release` and `master`.
+* All new PRs are created and merged against `master`
+* All versions are created from the `active_release` branch
+* Once merged to master, changes are "cherry-picked" (via a hopefully soon to be automated process), to the `active_release` branch based on the judgement of the original PR author and maintainers.
+
+* We do not merge breaking api changes, as defined in [Rust RFC 1105](https://github.com/rust-lang/rfcs/blob/master/text/1105-api-evolution.md) to the `active_release`
+
+Please see the [original proposal](https://docs.google.com/document/d/1tMQ67iu8XyGGZuj--h9WQYB9inCk6c2sL_4xMTwENGc/edit?ts=60961758) document the rational of this change.
+
+## Release Branching
+We aim to release every other week from the `active_release` branch.
+
+Every other Monday, a maintainer proposes a minor (e.g. `4.1.0` to `4.2.0`) or patch (e.g `4.1.0` to `4.1.1`) release, depending on changes to the `active_release` in the previous 2 weeks, following the process beloe.
+
+If this release is approved by at least three PMC members, a new version from that tarball is released to crates.io later in the week.
+
+Apache Arrow in general does synchronized major releases every three months. The Rust implementation aims to do its major releases in the same time frame.
+
+# Release Mechanics
+
+This directory contains the scripts used to manage an Apache Arrow Release.
+
+# Process Overview
+As part of the Apache governance model, official releases consist of
+signed source tarballs approved by the PMC.
+
+We then use the code in the approved source tarball to release to
+crates.io, the Rust ecosystem's package manager.
+
+## Branching
+
+
+# Release Preparation
+
+# Change Log
+
+We create a `CHANGELOG.md` so our users know what has been changed between releases.
+
+The CHANGELOG is created automatically using
+[change_log.sh](https://github.com/apache/arrow-rs/blob/master/change_log.sh)
+
+This script creates a changelog using github issues and the
+labels associated with them.
+
+
+
+
+# Mechanics of creating a release
+
+## Prepare the release branch and tags
+
+First, ensure that `active_release` contains the content of the desired release. For minor and patch releases, no additional steps are needed.
+
+To prepare for *a major release*, change `active release` to point at the latest `master` with commands such as:
+
+```
+git checkout active_release
+git fetch apache
+git reset --hard apache/master
+git push -f
+```
+
+### Update CHANGELOG.md + Version
+
+Now prepare a PR to update `CHANGELOG.md` and versions on `active_release` branch to reflect the planned release.
+
+See [#298](https://github.com/apache/arrow-rs/pull/298) for an example.
+
+Here are the commands used to prepare the 4.1.0 release:
+
+```bash
+git checkout active_release
+git pull
+git checkout -b make-release
+
+# manully edit ./dev/release/update_change_log.sh to reflect the release version
+# create the changelog
+CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log.sh
+# review change log / edit issues and labels if needed, rerun
+git commit -a -m 'Create changelog'
+
+# update versions
+sed -i '' -e 's/5.0.0-SNAPSHOT/4.1.0/g' `find . -name 'Cargo.toml'`
+git commit -a -m 'Update version'
+```
+
+Note that when reviewing the change log, rather than editing the
+`CHANGELOG.md`, it is preferred to update the issues and their labels
+(e.g. add `invalid` label to exclude them from release notes)
+
+
+## Prepare release candidate tarball
+
+(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)
+
+### Create git tag for the release:
+
+While the official release artifact is a signed tarball, we also tag the commit it was created for convenience and code archaeology.
+
+Using a string such as `4.0.1` as the `<version>`, create and push the tag thusly:
+
+```shell
+git fetch apache
+git tag <version> apache/active_release
+# push tag to apache
+git push apache <version>
+```
+
+### Pick an Release Candidate (RC) number
+
+Pick numbers in sequential order, with `0` for `rc1`, `1` for `rc1`, etc.
+
+### Create, sign, and upload tarball
+
+Run the `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:
+
+```shell
+./dev/release/create-tarball.sh 4.1.0 2
+```
+
+This script
+
+1. creates and uploads a release candidate tarball to the [arrow
+dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
+apache distribution svn server
+
+2. provide you an email template to
+send to dev@arrow.apache.org for release voting.
+
+
+### Vote on Release Candidate tarball
+
+Send the email output from the script to dev@arrow.apache.org. The email should look like
+
+```
+To: dev@arrow.apache.org
+Subject: [VOTE][RUST] Release Apache Arrow
+
+Hi,
+
+I would like to propose a release of Apache Arrow Rust
+Implementation, version 4.1.0.
+
+This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
+
+The proposed release tarball and signatures are hosted at [2].
+The changelog is located at [3].
+
+Please download, verify checksums and signatures, run the unit tests,
+and vote on the release.
+
+The vote will be open for at least 72 hours.
+
+[ ] +1 Release this as Apache Arrow Rust
+[ ] +0
+[ ] -1 Do not release this as Apache Arrow Rust because...
+
+[1]: https://github.com/apache/arrow-rs/tree/a5dd428f57e62db20a945e8b1895de91405958c4
+[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-4.1.0
+[3]: https://github.com/apache/arrow-rs/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
+```
+
+For the release to become "official" it needs at least three PMC members to vote +1 on it.
+
+
+#### Verifying Release Candidates
+
+There is a script in this repository which can be used to help `dev/release/verify-release-candidate.sh` assist the verification process. Run it like:
+
+```
+./dev/release/verify-release-candidate.sh 4.1.0 2
+```
+
+
+#### If the release is not approved
+
+If the release is not approved, fix whatever the problem is and try again with the next RC number
+
+
+### If the release is approved,
+
+Move tarball to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/arrow/arrow-4.1.0/, using the `release-tarball.sh` script:
+
+```shell
+./dev/release/release-tarball.sh 4.1.0 2
+```
+
+### Publish on Crates.io
+
+Only approved releases of the tarball should be published to
+crates.io, in order to conform to Apache Software Foundation
+governance standards.
+
+An Arrow committer can publish this crate after an official project release has
+been made to crates.io using the following instructions.
+
+Follow [these
+instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to
+create an account and login to crates.io before asking to be added as an owner
+of the [arrow crate](https://crates.io/crates/arrow).
+
+Download and unpack the official release tarball
+
+Verify that the Cargo.toml in the tarball contains the correct version
+(e.g. `version = "0.11.0"`) and then publish the crate with the
+following commands
+
+```shell
+(cd arrow && cargo publish)
+(cd arrow_flight && cargo publish)
+(cd parquet && cargo publish)
+(cd parquet_derive && cargo publish)
+```
diff --git a/dev/release/check-rat-report.py b/dev/release/check-rat-report.py
new file mode 100644
index 0000000..e30d72b
--- /dev/null
+++ b/dev/release/check-rat-report.py
@@ -0,0 +1,59 @@
+#!/usr/bin/python
+##############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+##############################################################################
+import fnmatch
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+if len(sys.argv) != 3:
+ sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" %
+ sys.argv[0])
+ sys.exit(1)
+
+exclude_globs_filename = sys.argv[1]
+xml_filename = sys.argv[2]
+
+globs = [line.strip() for line in open(exclude_globs_filename, "r")]
+
+tree = ET.parse(xml_filename)
+root = tree.getroot()
+resources = root.findall('resource')
+
+all_ok = True
+for r in resources:
+ approvals = r.findall('license-approval')
+ if not approvals or approvals[0].attrib['name'] == 'true':
+ continue
+ clean_name = re.sub('^[^/]+/', '', r.attrib['name'])
+ excluded = False
+ for g in globs:
+ if fnmatch.fnmatch(clean_name, g):
+ excluded = True
+ break
+ if not excluded:
+ sys.stdout.write("NOT APPROVED: %s (%s): %s\n" % (
+ clean_name, r.attrib['name'], approvals[0].attrib['name']))
+ all_ok = False
+
+if not all_ok:
+ sys.exit(1)
+
+print('OK')
+sys.exit(0)
diff --git a/dev/release/create-tarball.sh b/dev/release/create-tarball.sh
new file mode 100755
index 0000000..ab3e1d2
--- /dev/null
+++ b/dev/release/create-tarball.sh
@@ -0,0 +1,123 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+# This script creates a signed tarball in
+# dev/dist/apache-arrow-rs-<version>-<sha>.tar.gz and uploads it to
+# the "dev" area of the dist.apache.arrow repository and prepares an
+# email for sending to the dev@arrow.apache.org list for a formal
+# vote.
+#
+# See release/README.md for full release instructions
+#
+# Requirements:
+#
+# 1. gpg setup for signing and have uploaded your public
+# signature to https://pgp.mit.edu/
+#
+# 2. Logged into the apache svn server with the appropriate
+# credentials
+#
+#
+# Based in part on 02-source.sh from apache/arrow
+#
+
+set -e
+
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)"
+
+if [ "$#" -ne 2 ]; then
+ echo "Usage: $0 <tag> <rc>"
+ echo "ex. $0 4.1.0 2"
+ exit
+fi
+
+tag=$1
+rc=$2
+
+release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})
+
+release=apache-arrow-rs-${tag}
+distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
+tarname=${release}.tar.gz
+tarball=${distdir}/${tarname}
+url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
+
+echo "Attempting to create ${tarball} from tag ${tag}"
+
+
+if [ -z "$release_hash" ]; then
+ echo "Cannot continue: unknown git tag: $tag"
+fi
+
+echo "Draft email for dev@arrow.apache.org mailing list"
+echo ""
+echo "---------------------------------------------------------"
+cat <<MAIL
+To: dev@arrow.apache.org
+Subject: [VOTE][RUST] Release Apache Arrow Rust ${version}
+
+Hi,
+
+I would like to propose a release of Apache Arrow Rust Implementation, version ${tag}.
+
+This release candidate is based on commit: ${release_hash} [1]
+
+The proposed release tarball and signatures are hosted at [2].
+
+The changelog is located at [3].
+
+Please download, verify checksums and signatures, run the unit tests,
+and vote on the release.
+
+The vote will be open for at least 72 hours.
+
+[ ] +1 Release this as Apache Arrow Rust ${version}
+[ ] +0
+[ ] -1 Do not release this as Apache Arrow Rust ${version} because...
+
+[1]: https://github.com/apache/arrow-rs/tree/${release_hash}
+[2]: ${url}
+[3]: https://github.com/apache/arrow-rs/blob/${release_hash}/CHANGELOG.md
+MAIL
+echo "---------------------------------------------------------"
+
+
+# create <tarball> containing the files in git at $release_hash
+# the files in the tarball are prefixed with {tag} (e.g. 4.0.1)
+mkdir -p ${distdir}
+(cd "${SOURCE_TOP_DIR}" && git archive ${release_hash} --prefix ${release}/ | gzip > ${tarball})
+
+echo "Running rat license checker on ${tarball}"
+${SOURCE_DIR}/run-rat.sh ${tarball}
+
+echo "Signing tarball and creating checksums"
+gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
+# create signing with relative path of tarball
+# so that they can be verified with a command such as
+# shasum --check apache-arrow-rs-4.1.0-rc2.tar.gz.sha512
+(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
+(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512
+
+echo "Uploading to apache dist/dev to ${url}"
+svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
+svn add ${distdir}
+svn ci -m "Apache Arrow Rust ${tag} ${rc}" ${distdir}
diff --git a/dev/release/release-tarball.sh b/dev/release/release-tarball.sh
new file mode 100755
index 0000000..74fcad7
--- /dev/null
+++ b/dev/release/release-tarball.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# This script copies a tarball from the "dev" area of the
+# dist.apache.arrow repository to the "release" area
+#
+# This script should only be run after the release has been approved
+# by the arrow PMC committee.
+#
+# See release/README.md for full release instructions
+#
+# Based in part on post-01-upload.sh from apache/arrow
+
+
+set -e
+set -u
+
+if [ "$#" -ne 2 ]; then
+ echo "Usage: $0 <version> <rc-num>"
+ echo "ex. $0 4.1.0 2"
+ exit
+fi
+
+version=$1
+rc=$2
+
+tmp_dir=tmp-apache-arrow-dist
+
+echo "Recreate temporary directory: ${tmp_dir}"
+rm -rf ${tmp_dir}
+mkdir -p ${tmp_dir}
+
+echo "Clone dev dist repository"
+svn \
+ co \
+ https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-${version}-rc${rc} \
+ ${tmp_dir}/dev
+
+echo "Clone release dist repository"
+svn co https://dist.apache.org/repos/dist/release/arrow ${tmp_dir}/release
+
+echo "Copy ${version}-rc${rc} to release working copy"
+release_version=arrow-${version}
+mkdir -p ${tmp_dir}/release/${release_version}
+cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/
+svn add ${tmp_dir}/release/${release_version}
+
+echo "Commit release"
+svn ci -m "Apache Arrow Rust ${version}" ${tmp_dir}/release
+
+echo "Clean up"
+rm -rf ${tmp_dir}
+
+echo "Success! The release is available here:"
+echo " https://dist.apache.org/repos/dist/release/arrow/${release_version}"
diff --git a/change_log.sh b/dev/release/update_change_log.sh
similarity index 75%
rename from change_log.sh
rename to dev/release/update_change_log.sh
index 3d9b57c..a11f966 100755
--- a/change_log.sh
+++ b/dev/release/update_change_log.sh
@@ -18,9 +18,22 @@
# under the License.
#
+# invokes the changelog generator with the config located in
+# arrow-rs/.github_changelog_generator
+#
+# Usage:
+# CHANGELOG_GITHUB_TOKEN=<TOKEN> ./update_change_log.sh
+
+set -e
+
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)"
+
+pushd ${SOURCE_TOP_DIR}
docker run -it --rm -e CHANGELOG_GITHUB_TOKEN=$CHANGELOG_GITHUB_TOKEN -v "$(pwd)":/usr/local/src/your-app githubchangeloggenerator/github-changelog-generator \
--user apache \
--project arrow-rs \
--since-commit 2021-04-20 \
- --future-release 4.0.1
+ --future-release 4.1.0
+
sed -i "s/\\\n/\n\n/" CHANGELOG.md