[GitHub] [arrow-datafusion] houqp commented on a change in pull request #875: document release process

[GitBox <gi...@apache.org>]

houqp commented on a change in pull request #875:
URL: https://github.com/apache/arrow-datafusion/pull/875#discussion_r688899751



##########
File path: dev/release/README.md
##########
@@ -0,0 +1,301 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Release Process
+
+## Sub-projects
+
+The Datafusion repo contains 3 different releasable sub-projects: Datafusion, Ballista and Datafusion python binding.
+
+We use Datafusion release to drive the release for the other sub-projects. As a
+result, Datafusion version bump is required for every release while version
+bumps for the Python binding and Ballista are optional. In other words, we can
+release a new version of Datafusion without releasing a new version of the
+Python binding or Ballista. On the other hand, releasing a new version of the
+Python binding or Ballista always requires a new Datafusion version release.
+
+## Branching
+
+Datafusion currently only releases from the `master` branch. Given the project
+is still in early development state, we are not maintaining an active stable
+release backport branch.
+
+## Prerequisite
+
+- Have upstream git repo `git@github.com:apache/arrow-datafusion.git` add as git remote `apache`.
+- Created a peronal access token in Github for changelog automation script.
+  - Github PAT should be created with `repo` access
+- Make sure your signing key is added to the following files in SVN:
+  - https://dist.apache.org/repos/dist/dev/arrow/KEYS
+  - https://dist.apache.org/repos/dist/release/arrow/KEYS
+
+## Process Overview
+
+As part of the Apache governance model, official releases consist of signed
+source tarballs approved by the PMC.
+
+We then use the code in the approved source tarball to release to crates.io and
+PyPI.
+
+### Change Log
+
+We maintain `CHANGELOG.md` for each sub project so our users know what has been
+changed between releases.
+
+The CHANGELOG is managed automatically using
+[update_change_log.sh](https://github.com/apache/arrow-datafusion/blob/master/dev/release/update_change_log.sh)
+
+This script creates a changelog using github PRs and issues based on the labels
+associated with them.
+
+## Prepare release comimts and PR
+
+Prepare a PR to update `CHANGELOG.md` and versions to reflect the planned
+release.
+
+See [#801](https://github.com/apache/arrow-datafusion/pull/801) for an example.
+
+Here are the commands that could be used to prepare the `5.1.0` release:
+
+### Update Version
+
+Checkout the master commit to be released
+
+```
+git fetch apache
+git checkout apache/master
+```
+
+Update datafusion version in `datafusion/Cargo.toml` to `5.1.0`.
+
+If there is new ballista release, update versions in ballista Cargo.tomls, run
+
+```
+./dev/update_ballista_versions.py 0.5.0
+```
+
+If there is new datafusion python binding release, update versions in
+`./python/Cargo.toml`.
+
+Lastly commit the version change:
+
+```
+git commit -a -m 'Update version'
+```
+
+### Update CHANGELOG.md
+
+Create local release rc tags:
+
+```
+git tag -f 5.1.0-rc-local
+# if there is ballista release
+git tag -f ballista-0.5.0-rc-local
+# if there is python binding release
+git tag -f python-0.3.0-rc-local
+```
+
+Manully edit the previous release version tag in
+`dev/release/update_change_log-{ballista,datafusion,python}.sh`. Commits
+between the previous verstion tag and the new rc tag will be used to
+populate the changelog content.
+
+```bash
+# create the changelog
+CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log-all.sh
+# review change log / edit issues and labels if needed, rerun until you are happy with the result
+git commit -a -m 'Create changelog for release'
+```
+
+Note that when reviewing the change log, rather than editing the
+`CHANGELOG.md`, it is preferred to update the issues and their labels.
+
+You can add `invalid` or `development-process` label to exclude items from
+release notes. Add `datafusion`, `ballista` and `python` labels to group items
+into each sub-project's change log.
+
+Send a PR to get these changes merged into `master` branch. If new commits that
+could change the change log content landed in the `master` branch before you
+could merge the PR, you need to rerun the changelog update script to regenerate
+the changelog and update the PR accordingly.
+
+## Prepare release candidate tarball
+
+After the PR gets merged, you are ready to create a releaes tarball from the
+merged commit.
+
+(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)
+
+### Pick an Release Candidate (RC) number
+
+Pick numbers in sequential order, with `0` for `rc0`, `1` for `rc1`, etc.
+
+### Create git tag for the release:
+
+While the official release artifact is a signed tarball, we also tag the commit it was created for convenience and code archaeology.
+
+Using a string such as `5.1.0` as the `<version>`, create and push the tag thusly:
+
+```shell
+git fetch apache
+git tag <version>-<rc> apache/master
+# push tag to Github remote
+git push apache <version>
+```
+
+### Create, sign, and upload tarball
+
+Run `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:
+
+```shell
+./dev/release/create-tarball.sh 5.1.0 0
+```
+
+The `create-tarball.sh` script
+
+1. creates and uploads a release candidate tarball to the [arrow
+   dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
+   apache distribution svn server
+
+2. provide you an email template to
+   send to dev@arrow.apache.org for release voting.
+
+### Vote on Release Candidate tarball
+
+Send the email output from the script to dev@arrow.apache.org. The email should look like
+
+```
+To: dev@arrow.apache.org
+Subject: [VOTE][RUST][Datafusion] Release Apache Arrow Datafusion 5.1.0 RC0
+
+Hi,
+
+I would like to propose a release of Apache Arrow Datafusion Implementation,
+version 5.1.0.
+
+This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
+The proposed release tarball and signatures are hosted at [2].
+The changelog is located at [3].
+
+Please download, verify checksums and signatures, run the unit tests,
+and vote on the release.
+
+The vote will be open for at least 72 hours.
+
+[ ] +1 Release this as Apache Arrow Datafusion 5.1.0
+[ ] +0
+[ ] -1 Do not release this as Apache Arrow Datafusion 5.1.0 because...
+
+[1]: https://github.com/apache/arrow-datafusion/tree/a5dd428f57e62db20a945e8b1895de91405958c4
+[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-datafusion-5.1.0
+[3]: https://github.com/apache/arrow-datafusion/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
+```
+
+For the release to become "official" it needs at least three PMC members to vote +1 on it.
+
+### Verifying Release Candidates
+
+The `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:
+
+```
+./dev/release/verify-release-candidate.sh 5.1.0 0
+```
+
+#### If the release is not approved
+
+If the release is not approved, fix whatever the problem is and try again with the next RC number
+
+### If the release is approved,
+
+Move tarball to the release location in SVN, e.g.
+https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-4.1.0/, using
+the `release-tarball.sh` script:
+
+```shell
+./dev/release/release-tarball.sh 5.1.0 0
+```
+
+Congratulations! The release is now offical!
+
+## Finalize the release
+
+### Create release git tags
+
+Tag the same release candidate commit with the final release tag
+
+```
+git co apache/5.1.0-RC0
+git tag 5.1.0
+git push 5.1.0
+```
+
+If there is ballista release, also push the ballista tag
+
+```
+git tag ballista-0.5.0
+git push ballista-0.5.0
+```
+
+If there is datafusion python binding release, also push the python tag
+
+```
+git tag python-0.3.0
+git push python-0.3.0
+```
+
+### Publish on Crates.io
+
+Only approved releases of the tarball should be published to
+crates.io, in order to conform to Apache Software Foundation
+governance standards.
+
+An Arrow committer can publish this crate after an official project release has
+been made to crates.io using the following instructions.
+
+Follow [these
+instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to
+create an account and login to crates.io before asking to be added as an owner
+of the following crates:
+
+- [datafusion](https://crates.io/crates/datafusion)
+- [ballista-core](https://crates.io/crates/ballista-core)
+- [ballista-executor](https://crates.io/crates/ballista-executor)
+- [ballista-scheduler](https://crates.io/crates/ballista-scheduler)
+
+Download and unpack the official release tarball
+
+Verify that the Cargo.toml in the tarball contains the correct version
+(e.g. `version = "5.1.0"`) and then publish the crate with the
+following commands
+
+```shell
+(cd datafusion && cargo publish)
+```
+
+If there is ballista release, run
+
+```shell
+(cd ballista/rust/core && cargo publish)
+(cd ballista/rust/executor && cargo publish)
+(cd ballista/rust/scheduler && cargo publish)
+```
+
+### Publish on PyPI
+
+TODO

Review comment:
       @jorgecarleitao Are you using `maturin` or `twine` to do the PyPI release?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org