You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2025/01/16 00:54:54 UTC
(datafusion-ballista) branch main updated: update release scripts to release to Apache DataFusion not Apache Arrow (#1163)
This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new 9c9ca0d3 update release scripts to release to Apache DataFusion not Apache Arrow (#1163)
9c9ca0d3 is described below
commit 9c9ca0d3342a17d5e83a11b331dfa332c8d790c3
Author: Andy Grove <ag...@apache.org>
AuthorDate: Wed Jan 15 17:54:50 2025 -0700
update release scripts to release to Apache DataFusion not Apache Arrow (#1163)
* update release scripts to release to Apache DataFusion not Apache Arrow
* more updates
---
dev/release/README.md | 84 +++++++++++----------------------
dev/release/create-tarball.sh | 35 +++++++-------
dev/release/release-tarball.sh | 16 +++----
dev/release/verify-release-candidate.sh | 30 ++++++------
4 files changed, 67 insertions(+), 98 deletions(-)
diff --git a/dev/release/README.md b/dev/release/README.md
index db19a982..51b4ef0d 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -60,9 +60,9 @@ Although some tasks can only be performed by a PMC member, many tasks can be per
### Post-Release
-| Task | Role Required |
-| ------------------------------------------------------- | ------------- |
-| Create PR against arrow-site with updated documentation | None |
+| Task | Role Required |
+|--------------------------------------------------------------| ------------- |
+| Create PR against datafusion-site with updated documentation | None |
## Detailed Guide
@@ -71,7 +71,7 @@ Although some tasks can only be performed by a PMC member, many tasks can be per
- You will need a GitHub Personal Access Token with "repo" access. Follow
[these instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
to generate one if you do not already have one.
-- Have upstream git repo `git@github.com:apache/arrow-ballista.git` add as git remote `apache`.
+- Have upstream git repo `git@github.com:apache/datafusion-ballista.git` add as git remote `apache`.
### Preparing the `main` Branch
@@ -102,7 +102,7 @@ pip3 install PyGitHub
Run the following command to generate the changelog content.
```bash
-$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py apache/arrow-ballista 0.11.0 HEAD > 0.12.0.md
+$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py apache/datafusion-ballista 0.11.0 HEAD > 0.12.0.md
```
This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for
@@ -121,9 +121,9 @@ This process is not fully automated, so there are some additional manual steps:
- Add the following content (copy from the previous version's changelog and update as appropriate:
```
-## [0.12.0](https://github.com/apache/arrow-ballista/tree/0.12.0) (2024-01-14)
+## [0.12.0](https://github.com/apache/datafusion-ballista/tree/0.12.0) (2024-01-14)
-[Full Changelog](https://github.com/apache/arrow-ballista/compare/0.11.0...0.12.0)
+[Full Changelog](https://github.com/apache/datafusion-ballista/compare/0.11.0...0.12.0)
```
Send a PR to get these changes merged into the release branch (e.g. `branch-0.12`). If new commits that could change the
@@ -157,16 +157,16 @@ git push apache <version>
### Create, sign, and upload artifacts
- Make sure your signing key is added to the following files in SVN:
- - https://dist.apache.org/repos/dist/dev/arrow/KEYS
- - https://dist.apache.org/repos/dist/release/arrow/KEYS
+ - https://dist.apache.org/repos/dist/dev/datafusion/KEYS
+ - https://dist.apache.org/repos/dist/release/datafusion/KEYS
See instructions at https://infra.apache.org/release-signing.html#generate for generating keys.
Committers can add signing keys in Subversion client with their ASF account. e.g.:
```bash
-$ svn co https://dist.apache.org/repos/dist/dev/arrow
-$ cd arrow
+$ svn co https://dist.apache.org/repos/dist/dev/datafusion
+$ cd datafusion
$ editor KEYS
$ svn ci KEYS
```
@@ -186,43 +186,17 @@ GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 0.11.0 1
The `create-tarball.sh` script
-1. creates and uploads all release candidate artifacts to the [arrow
- dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
+1. creates and uploads all release candidate artifacts to the [datafusion
+ dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the
apache distribution svn server
2. provide you an email template to
- send to dev@arrow.apache.org for release voting.
+ send to dev@datafusion.apache.org for release voting.
### Vote on Release Candidate artifacts
-Send the email output from the script to dev@arrow.apache.org. The email should look like
+Send the email output from the script to dev@datafusion.apache.org.
-```
-To: dev@arrow.apache.org
-Subject: [VOTE][Ballista] Release Apache Arrow Ballista 0.8.0 RC0
-
-Hi,
-
-I would like to propose a release of Apache Arrow Ballista Implementation,
-version 0.8.0.
-
-This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
-The proposed release artifacts and signatures are hosted at [2].
-The changelog is located at [3].
-
-Please download, verify checksums and signatures, run the unit tests,
-and vote on the release.
-
-The vote will be open for at least 72 hours.
-
-[ ] +1 Release this as Apache Arrow Ballista 0.8.0
-[ ] +0
-[ ] -1 Do not release this as Apache Arrow Ballista 0.8.0 because...
-
-[1]: https://github.com/apache/arrow-ballista/tree/a5dd428f57e62db20a945e8b1895de91405958c4
-[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-0.8.0
-[3]: https://github.com/apache/arrow-ballista/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
-```
For the release to become "official" it needs at least three PMC members to vote +1 on it.
@@ -246,7 +220,7 @@ NOTE: steps in this section can only be done by PMC members.
### After the release is approved
Move artifacts to the release location in SVN, e.g.
-https://dist.apache.org/repos/dist/release/arrow/arrow-ballista-0.8.0/, using
+https://dist.apache.org/repos/dist/release/datafusion/datafusion-ballista-0.8.0/, using
the `release-tarball.sh` script:
```shell
@@ -271,7 +245,7 @@ Only approved releases of the tarball should be published to
crates.io, in order to conform to Apache Software Foundation
governance standards.
-An Arrow committer can publish this crate after an official project release has
+A DataFusion committer can publish this crate after an official project release has
been made to crates.io using the following instructions.
Follow [these
@@ -311,11 +285,11 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg
Pushing a release tag causes Docker images to be published.
-Images can be found at [https://github.com/apache/arrow-ballista/pkgs/container/arrow-ballista-standalone](https://github.com/apache/arrow-ballista/pkgs/container/arrow-ballista-standalone)
+Images can be found at [https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone](https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone)
### Call the vote
-Call the vote on the Arrow dev list by replying to the RC voting thread. The
+Call the vote on the DataFusion dev list by replying to the RC voting thread. The
reply should have a new subject constructed by adding `[RESULT]` prefix to the
old subject line.
@@ -328,8 +302,8 @@ with the release verification.
### Add the release to Apache Reporter
-Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with `RS-BALLISTA-`,
-for example `RS-BALLISTA-0.9.0`.
+Add the release to https://reporter.apache.org/addrelease.html?datafusion with a version name prefixed with `BALLISTA-`,
+for example `BALLISTA-0.9.0`.
The release information is used to generate a template for a board report (see example
[here](https://github.com/apache/arrow/pull/14357)).
@@ -346,13 +320,13 @@ Release candidates should be deleted once the release is published.
Get a list of Ballista release candidates:
```bash
-svn ls https://dist.apache.org/repos/dist/dev/arrow | grep ballista
+svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep ballista
```
Delete a release candidate:
```bash
-svn delete -m "delete old Ballista RC" https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-0.8.0-rc1/
+svn delete -m "delete old Ballista RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ballista-0.8.0-rc1/
```
#### Deleting old releases from `release` svn
@@ -362,13 +336,13 @@ Only the latest release should be available. Delete old releases after publishin
Get a list of Ballista releases:
```bash
-svn ls https://dist.apache.org/repos/dist/release/arrow | grep ballista
+svn ls https://dist.apache.org/repos/dist/release/datafusion | grep ballista
```
Delete a release:
```bash
-svn delete -m "delete old Ballista release" https://dist.apache.org/repos/dist/release/arrow/arrow-ballista-0.8.0
+svn delete -m "delete old Ballista release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-ballista-0.8.0
```
### Optional: Write a blog post announcing the release
@@ -384,11 +358,7 @@ git shortlog -sn 0.11.0..0.10.0 ballista ballista-cli examples | wc -l
```
Once there is consensus on the contents of the post, create a PR to add a blog post to the
-[arrow-site](https://github.com/apache/arrow-site) repository. Note that there is no need for a formal
+[datafusion-site](https://github.com/apache/datafusion-site) repository. Note that there is no need for a formal
PMC vote on the blog post contents since this isn't considered to be a "release".
-Here is an example blog post PR:
-
-- https://github.com/apache/arrow-site/pull/217
-
-Once the PR is merged, a GitHub action will publish the new blog post to https://arrow.apache.org/blog/.
+Once the PR is merged, a GitHub action will publish the new blog post to https://datafusion.apache.org/blog/.
diff --git a/dev/release/create-tarball.sh b/dev/release/create-tarball.sh
index f5a43727..3865e8a4 100755
--- a/dev/release/create-tarball.sh
+++ b/dev/release/create-tarball.sh
@@ -21,9 +21,9 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh
# This script creates a signed tarball in
-# dev/dist/apache-arrow-ballista-<version>-<sha>.tar.gz and uploads it to
-# the "dev" area of the dist.apache.arrow repository and prepares an
-# email for sending to the dev@arrow.apache.org list for a formal
+# dev/dist/apache-datafusion-ballista-<version>-<sha>.tar.gz and uploads it to
+# the "dev" area of the dist.apache.datafusion repository and prepares an
+# email for sending to the dev@datafusion.apache.org list for a formal
# vote.
#
# See release/README.md for full release instructions
@@ -65,26 +65,25 @@ tag="${version}-rc${rc}"
echo "Attempting to create ${tarball} from tag ${tag}"
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})
-release=apache-arrow-ballista-${version}
+release=apache-datafusion-ballista-${version}
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
tarname=${release}.tar.gz
tarball=${distdir}/${tarname}
-url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
+url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"
if [ -z "$release_hash" ]; then
echo "Cannot continue: unknown git tag: ${tag}"
fi
-echo "Draft email for dev@arrow.apache.org mailing list"
+echo "Draft email for dev@datafusion.apache.org mailing list"
echo ""
echo "---------------------------------------------------------"
cat <<MAIL
-To: dev@arrow.apache.org
-Subject: [VOTE][RUST][Ballista] Release Apache Arrow Ballista ${version} RC${rc}
+To: dev@datafusion.apache.org
+Subject: [VOTE][RUST][Ballista] Release Apache DataFusion Ballista ${version} RC${rc}
Hi,
-I would like to propose a release of Apache Arrow Ballista Implementation,
-version ${version}.
+I would like to propose a release of Apache DataFusion Ballista version ${version}.
This release candidate is based on commit: ${release_hash} [1]
The proposed release tarball and signatures are hosted at [2].
@@ -96,15 +95,15 @@ on the release. The vote will be open for at least 72 hours.
Only votes from PMC members are binding, but all members of the community are
encouraged to test the release and vote with "(non-binding)".
-The standard verification procedure is documented at https://github.com/apache/arrow-ballista/blob/main/dev/release/README.md#verifying-release-candidates.
+The standard verification procedure is documented at https://github.com/apache/datafusion-ballista/blob/main/dev/release/README.md#verifying-release-candidates.
-[ ] +1 Release this as Apache Arrow Ballista ${version}
+[ ] +1 Release this as Apache DataFusion Ballista ${version}
[ ] +0
-[ ] -1 Do not release this as Apache Arrow Ballista ${version} because...
+[ ] -1 Do not release this as Apache DataFusion Ballista ${version} because...
-[1]: https://github.com/apache/arrow-ballista/tree/${release_hash}
+[1]: https://github.com/apache/datafusion-ballista/tree/${release_hash}
[2]: ${url}
-[3]: https://github.com/apache/arrow-ballista/blob/${release_hash}/CHANGELOG.md
+[3]: https://github.com/apache/datafusion-ballista/blob/${release_hash}/CHANGELOG.md
MAIL
echo "---------------------------------------------------------"
@@ -121,12 +120,12 @@ echo "Signing tarball and creating checksums"
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
# create signing with relative path of tarball
# so that they can be verified with a command such as
-# shasum --check apache-arrow-ballista-4.1.0-rc2.tar.gz.sha512
+# shasum --check apache-datafusion-ballista-4.1.0-rc2.tar.gz.sha512
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512
echo "Uploading to apache dist/dev to ${url}"
-svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
+svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
svn add ${distdir}
-svn ci -m "Apache Arrow Ballista ${version} ${rc}" ${distdir}
+svn ci -m "Apache DataFusion Ballista ${version} ${rc}" ${distdir}
diff --git a/dev/release/release-tarball.sh b/dev/release/release-tarball.sh
index de1ee333..c51d3cb2 100755
--- a/dev/release/release-tarball.sh
+++ b/dev/release/release-tarball.sh
@@ -21,10 +21,10 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/release-tarball.sh
# This script copies a tarball from the "dev" area of the
-# dist.apache.arrow repository to the "release" area
+# dist.apache.datafusion repository to the "release" area
#
# This script should only be run after the release has been approved
-# by the arrow PMC committee.
+# by the DataFusion PMC committee.
#
# See release/README.md for full release instructions
#
@@ -43,7 +43,7 @@ fi
version=$1
rc=$2
-tmp_dir=tmp-apache-arrow-ballista-dist
+tmp_dir=tmp-apache-datafusion-ballista-dist
echo "Recreate temporary directory: ${tmp_dir}"
rm -rf ${tmp_dir}
@@ -52,23 +52,23 @@ mkdir -p ${tmp_dir}
echo "Clone dev dist repository"
svn \
co \
- https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-${version}-rc${rc} \
+ https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ballista-${version}-rc${rc} \
${tmp_dir}/dev
echo "Clone release dist repository"
-svn co https://dist.apache.org/repos/dist/release/arrow ${tmp_dir}/release
+svn co https://dist.apache.org/repos/dist/release/datafusion ${tmp_dir}/release
echo "Copy ${version}-rc${rc} to release working copy"
-release_version=arrow-ballista-${version}
+release_version=datafusion-ballista-${version}
mkdir -p ${tmp_dir}/release/${release_version}
cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/
svn add ${tmp_dir}/release/${release_version}
echo "Commit release"
-svn ci -m "Apache Arrow Ballista ${version}" ${tmp_dir}/release
+svn ci -m "Apache DataFusion Ballista ${version}" ${tmp_dir}/release
echo "Clean up"
rm -rf ${tmp_dir}
echo "Success! The release is available here:"
-echo " https://dist.apache.org/repos/dist/release/arrow/${release_version}"
+echo " https://dist.apache.org/repos/dist/release/datafusion/${release_version}"
diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index da1fcd10..c4c5bdb3 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -32,8 +32,8 @@ set -x
set -o pipefail
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
-ARROW_DIR="$(dirname $(dirname ${SOURCE_DIR}))"
-ARROW_DIST_URL='https://dist.apache.org/repos/dist/dev/arrow'
+DATAFUSION_DIR="$(dirname $(dirname ${SOURCE_DIR}))"
+DATAFUSION_DIST_URL='https://dist.apache.org/repos/dist/dev/datafusion'
download_dist_file() {
curl \
@@ -41,11 +41,11 @@ download_dist_file() {
--show-error \
--fail \
--location \
- --remote-name $ARROW_DIST_URL/$1
+ --remote-name $DATAFUSION_DIST_URL/$1
}
download_rc_file() {
- download_dist_file apache-arrow-ballista-${VERSION}-rc${RC_NUMBER}/$1
+ download_dist_file apache-datafusion-ballista-${VERSION}-rc${RC_NUMBER}/$1
}
import_gpg_keys() {
@@ -89,19 +89,19 @@ verify_dir_artifact_signatures() {
setup_tempdir() {
cleanup() {
if [ "${TEST_SUCCESS}" = "yes" ]; then
- rm -fr "${ARROW_TMPDIR}"
+ rm -fr "${DATAFUSION_TMPDIR}"
else
- echo "Failed to verify release candidate. See ${ARROW_TMPDIR} for details."
+ echo "Failed to verify release candidate. See ${DATAFUSION_TMPDIR} for details."
fi
}
- if [ -z "${ARROW_TMPDIR}" ]; then
- # clean up automatically if ARROW_TMPDIR is not defined
- ARROW_TMPDIR=$(mktemp -d -t "$1.XXXXX")
+ if [ -z "${DATAFUSION_TMPDIR}" ]; then
+ # clean up automatically if DATAFUSION_TMPDIR is not defined
+ DATAFUSION_TMPDIR=$(mktemp -d -t "$1.XXXXX")
trap cleanup EXIT
else
# don't clean up automatically
- mkdir -p "${ARROW_TMPDIR}"
+ mkdir -p "${DATAFUSION_TMPDIR}"
fi
}
@@ -124,7 +124,7 @@ test_source_distribution() {
# Clone testing repositories if not cloned already
git clone https://github.com/apache/arrow-testing.git arrow-testing-data
git clone https://github.com/apache/parquet-testing.git parquet-testing-data
- export ARROW_TEST_DATA=$PWD/arrow-testing-data/data
+ export DATAFUSION_TEST_DATA=$PWD/arrow-testing-data/data
export PARQUET_TEST_DATA=$PWD/parquet-testing-data/data
# TODO: enable this eventually so that cargo test will check benchmark query results
@@ -150,11 +150,11 @@ test_source_distribution() {
TEST_SUCCESS=no
-setup_tempdir "arrow-${VERSION}"
-echo "Working in sandbox ${ARROW_TMPDIR}"
-cd ${ARROW_TMPDIR}
+setup_tempdir "datafusion-${VERSION}"
+echo "Working in sandbox ${DATAFUSION_TMPDIR}"
+cd ${DATAFUSION_TMPDIR}
-dist_name="apache-arrow-ballista-${VERSION}"
+dist_name="apache-datafusion-ballista-${VERSION}"
import_gpg_keys
fetch_archive ${dist_name}
tar xf ${dist_name}.tar.gz
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org