You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/06 17:06:49 UTC
[GitHub] [airflow] potiuk opened a new pull request #11310: Merge Airflow and Backport Packages preparation instructions
potiuk opened a new pull request #11310:
URL: https://github.com/apache/airflow/pull/11310
This commit extracts common parts of Apache Airflow package
preparation and Backport Packages preparation.
Common parts were extracted as prerequisites, the release process
has been put in chronological order, some details about preparing
backport packages have been moved to a separate README.md
in the Backport Packages to not confuse release instructions
with tool instructions.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501940320
##########
File path: BREEZE.rst
##########
@@ -2150,6 +2155,11 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
+ --install-wheels
Review comment:
It's automated output from --help-all in breeze - and this flag is valid for two command - enter breeze and start-airflow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
>. -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-704872877
I'd really love more people get involved in it :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-704425757
cc: @sergioesca4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501600897
##########
File path: backport_packages/README.md
##########
@@ -0,0 +1,173 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Backport packages
+
+# What the backport packages are
+
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
+
+The release manager prepares backport packages separately from the main Airflow Release, using
+`breeze` commands and accompanying scripts. This document provides an overview of the command line tools
+needed to prepare backport packages.
+
+# Content of the release notes
+
+Each of the backport packages contains Release notes in the form of the README.md file that is
+automatically generated from history of the changes and code of the provider.
+
+The script generates all the necessary information:
+
+* summary of requirements for each backport package
+* list of dependencies (including extras to install them) when package
+ depends on other providers packages
+* table of new hooks/operators/sensors/protocols/secrets
+* table of moved hooks/operators/sensors/protocols/secrets with the
+ information where they were moved from
+* changelog of all the changes to the provider package (this will be
+ automatically updated with an incremental changelog whenever we decide to
+ release separate packages.
Review comment:
Missing closing bracket
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501600897
##########
File path: backport_packages/README.md
##########
@@ -0,0 +1,173 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# Backport packages
+
+# What the backport packages are
+
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
+
+The release manager prepares backport packages separately from the main Airflow Release, using
+`breeze` commands and accompanying scripts. This document provides an overview of the command line tools
+needed to prepare backport packages.
+
+# Content of the release notes
+
+Each of the backport packages contains Release notes in the form of the README.md file that is
+automatically generated from history of the changes and code of the provider.
+
+The script generates all the necessary information:
+
+* summary of requirements for each backport package
+* list of dependencies (including extras to install them) when package
+ depends on other providers packages
+* table of new hooks/operators/sensors/protocols/secrets
+* table of moved hooks/operators/sensors/protocols/secrets with the
+ information where they were moved from
+* changelog of all the changes to the provider package (this will be
+ automatically updated with an incremental changelog whenever we decide to
+ release separate packages.
Review comment:
Missing closing bracket
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
Review comment:
```suggestion
via [Official Apache Download sources](https://downloads.apache.org/airflow/backport-providers/).
```
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
-* -bin-tar.gz + .asc + .sha512
-* -source.tar.gz + .asc + .sha512
-* -.whl + .asc + .sha512
+There are also 50+ convenience packages released as "apache-airflow-backport-providers" separately in
+PyPI. You can find them all by [PyPI query](https://pypi.org/search/?q=apache-airflow-backport-providers)
-As a PMC you should be able to clone the SVN repository:
+The document describes the process of releasing both - official source packages and convenience
+packages for Backport Provider Packages.
-```bash
-svn co https://dist.apache.org/repos/dist/dev/airflow
-```
+# Prerequisites for the release manager preparing the release
-Or update it if you already checked it out:
+The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
+available in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the release manager
+can ask PMC to perform that final step of release.
-```bash
-svn update .
-```
+## Upload Public keys to id.apache.org
-## Verifying the licences
+Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
-This can be done with the Apache RAT tool.
+- Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of that file. Upload your GPG public key to https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache credentials, paste your fingerprint into the pgp fingerprint field and hit save).
-* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
- the jar is inside)
-* Unpack the -source.tar.gz to a folder
-* Enter the folder and run the check (point to the place where you extracted the .jar)
+```shell script
+# Create PGP Key
+gpg --gen-key
-```bash
-java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
-```
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/airflow
+cd airflow
-## Verifying the signatures
-Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
+(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS
-You can import the whole KEYS file:
-```bash
-gpg --import KEYS
+# Commit the changes
+svn commit -m "Add PGP keys of Airflow developers"
```
-You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
-retrieves it from the default GPG keyserver
-[OpenPGP.org](https://keys.openpgp.org):
-
-```bash
-gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+See this for more detail on creating keys and what is required for signing releases.
-You should choose to import the key when asked.
+http://www.apache.org/dev/release-signing.html#basic-facts
-Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
-errors or timeouts. Many of the release managers also uploaded their keys to the
-[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+## Configure PyPI uploads
-```bash
-gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens.
+You can add and copy the tokens here:
-Once you have the keys, the signatures can be verified by running this:
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
-```bash
-for i in *.asc
-do
- echo "Checking $i"; gpg --verify `basename $i .sha512 `
-done
-```
-This should produce results similar to the below. The "Good signature from ..." is indication
-that the signatures are correct. Do not worry about the "not certified with a trusted signature"
-warning. Most of certificates used by release managers are self signed, that's why you get this
-warning. By importing the server in the previous step and importing it via ID from
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
-this is a valid Key already.
+Create a ~/.pypirc file:
-```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
-gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
-gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-```
+```shell script
+[distutils]
+index-servers =
+ pypi
+ pypitest
-## Verifying the SHA512 sum
+[pypi]
+username=__token__
+password=<API Upload Token>
-Run this:
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
-```bash
-for i in *.sha512
-do
- echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
-done
```
-You should get output similar to:
+Set proper permissions for the pypirc file:
+```shell script
+$ chmod 600 ~/.pypirc
```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
-Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
-```
-
-# Verifying if the release candidate "works" by Contributors
-This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
-actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
-is available on PyPI apart from SVN packages, so everyone should be able to install
-the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
-release candidate number 1,2,3,....).
+* Install twine if you do not have it already.
-```bash
-pip install apache-airflow==<VERSION>rc<X>
+```shell script
+pip install twine
```
-Optionally it can be followed with constraints
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
-Note that the constraints contain python version that you are installing it with.
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+- Confirm that airflow/version.py is set properly.
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Here is a typical scenario:
+- Install [twine](https://pypi.org/project/twine/) if you do not have it already (it can be done
+ in a separate virtual environment).
+ `pip install twine`
Review comment:
Duplicate of Line 185
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
-* -bin-tar.gz + .asc + .sha512
-* -source.tar.gz + .asc + .sha512
-* -.whl + .asc + .sha512
+There are also 50+ convenience packages released as "apache-airflow-backport-providers" separately in
+PyPI. You can find them all by [PyPI query](https://pypi.org/search/?q=apache-airflow-backport-providers)
-As a PMC you should be able to clone the SVN repository:
+The document describes the process of releasing both - official source packages and convenience
+packages for Backport Provider Packages.
-```bash
-svn co https://dist.apache.org/repos/dist/dev/airflow
-```
+# Prerequisites for the release manager preparing the release
-Or update it if you already checked it out:
+The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
+available in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the release manager
+can ask PMC to perform that final step of release.
-```bash
-svn update .
-```
+## Upload Public keys to id.apache.org
-## Verifying the licences
+Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
-This can be done with the Apache RAT tool.
+- Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of that file. Upload your GPG public key to https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache credentials, paste your fingerprint into the pgp fingerprint field and hit save).
-* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
- the jar is inside)
-* Unpack the -source.tar.gz to a folder
-* Enter the folder and run the check (point to the place where you extracted the .jar)
+```shell script
+# Create PGP Key
+gpg --gen-key
-```bash
-java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
-```
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/airflow
+cd airflow
-## Verifying the signatures
-Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
+(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS
-You can import the whole KEYS file:
-```bash
-gpg --import KEYS
+# Commit the changes
+svn commit -m "Add PGP keys of Airflow developers"
```
-You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
-retrieves it from the default GPG keyserver
-[OpenPGP.org](https://keys.openpgp.org):
-
-```bash
-gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+See this for more detail on creating keys and what is required for signing releases.
-You should choose to import the key when asked.
+http://www.apache.org/dev/release-signing.html#basic-facts
-Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
-errors or timeouts. Many of the release managers also uploaded their keys to the
-[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+## Configure PyPI uploads
-```bash
-gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens.
+You can add and copy the tokens here:
-Once you have the keys, the signatures can be verified by running this:
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
-```bash
-for i in *.asc
-do
- echo "Checking $i"; gpg --verify `basename $i .sha512 `
-done
-```
-This should produce results similar to the below. The "Good signature from ..." is indication
-that the signatures are correct. Do not worry about the "not certified with a trusted signature"
-warning. Most of certificates used by release managers are self signed, that's why you get this
-warning. By importing the server in the previous step and importing it via ID from
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
-this is a valid Key already.
+Create a ~/.pypirc file:
-```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
-gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
-gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-```
+```shell script
+[distutils]
+index-servers =
+ pypi
+ pypitest
-## Verifying the SHA512 sum
+[pypi]
+username=__token__
+password=<API Upload Token>
-Run this:
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
-```bash
-for i in *.sha512
-do
- echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
-done
```
-You should get output similar to:
+Set proper permissions for the pypirc file:
+```shell script
+$ chmod 600 ~/.pypirc
```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
-Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
-```
-
-# Verifying if the release candidate "works" by Contributors
-This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
-actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
-is available on PyPI apart from SVN packages, so everyone should be able to install
-the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
-release candidate number 1,2,3,....).
+* Install twine if you do not have it already.
-```bash
-pip install apache-airflow==<VERSION>rc<X>
+```shell script
+pip install twine
```
-Optionally it can be followed with constraints
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
-Note that the constraints contain python version that you are installing it with.
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+- Confirm that airflow/version.py is set properly.
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Here is a typical scenario:
+- Install [twine](https://pypi.org/project/twine/) if you do not have it already (it can be done
+ in a separate virtual environment).
+ `pip install twine`
-1. `./breeze --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres`
-2. `tmux`
-3. Hit Ctrl-B followed by "
-4. `airflow resetdb -y`
-5. if you want RBAC:
- * Change RBAC setting: `sed "s/rbac = False/rbac = True/" -i /root/airflow/airflow.cfg`
- * airflow resetdb -y
- * Run`airflow create_user -r Admin -u airflow -e airflow@apache.org -f Airflow -l User -p airflow
-6. `airflow scheduler`
-7. Ctrl-B "up-arrow"
-8. `airflow webserver`
+## Hardware used to prepare and verify the packages.
Review comment:
```suggestion
## Hardware used to prepare and verify the packages
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
```Dockerfile
FROM apache/airflow:1.10.12
RUN BACKPORT_RELEASE=2020.10.5rc1 \
&& pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
&& pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
RUN curl https://sdk.cloud.google.com | bash \
&& echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
&& echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
USER 0
RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
&& KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
&& curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
USER ${AIRFLOW_UID}
```
Feel free to modify this example to test your use cases.
To build an image build and run a shell, run:
```bash
docker build . -t my-airflow
docker run -ti \
--rm \
-v "$PWD/data:/opt/airflow/" \
-v "$PWD/keys/:/keys/" \
-p 8080:8080 \
-e AIRFLOW__CORE__LOAD_EXAMPLES=True \
my-airflow
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #11310:
URL: https://github.com/apache/airflow/pull/11310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501602658
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
Review comment:
```suggestion
via [Official Apache Download sources](https://downloads.apache.org/airflow/backport-providers/).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501536269
##########
File path: dev/README.md
##########
@@ -575,8 +706,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
Review comment:
@ashb @kaxil @mik-laj I've added links to instructions on how to test the RC in the email template.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501536269
##########
File path: dev/README.md
##########
@@ -575,8 +706,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
Review comment:
@ashb @kaxil @mik-laj I've added links to instructions on how to test the RC in the email template.
##########
File path: dev/README.md
##########
@@ -388,8 +341,11 @@ with INSTALL instructions.
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
-Only votes from PMC members are binding, but members of the community are
-encouraged to test the release and vote with "(non-binding)".
+Only votes from PMC members are binding, but the release manager should encourage members of the community
+to test the release and vote with "(non-binding)".
+
+The test procedure for PMCs and Contributors who would like to test this RC are described in
Review comment:
@ashb @kaxil @mik-laj -> also here, the link to test procedure for PMCs and contributors is added
##########
File path: BREEZE.rst
##########
@@ -2150,6 +2155,11 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
+ --install-wheels
Review comment:
It's automated output from --help-all in breeze - and this flag is valid for two command - enter breeze and start-airflow
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I incorporated your changes @mik-laj slightly modified it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501950831
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I incorporated your changes @mik-laj slightly modified it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-705408793
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501952278
##########
File path: dev/README.md
##########
@@ -211,364 +1091,295 @@ Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
```
-# Verifying if the release candidate "works" by Contributors
+### Verify if the Backport Packages release candidates "work" by Contributors
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
is available on PyPI apart from SVN packages, so everyone should be able to install
the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
release candidate number 1,2,3,....).
-```bash
-pip install apache-airflow==<VERSION>rc<X>
-```
-Optionally it can be followed with constraints
-
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
-Note that the constraints contain python version that you are installing it with.
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+#### Installing in your local virtualenv
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
+You have to make sure you have Airilow 1.10.* installed in your PIP virtualenv
+(the version you want to install providers with).
-```
-./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
```
-Once you install and run Airflow, you should perform any verification you see as necessary to check
-that the Airflow works as you expected.
+#### Installing with Breeze
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
-# Building an RC
-
-The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
+First copy all the provider packages .whl files to the `dist` folder.
-- Set environment variables
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
```
-# Set Version
-export VERSION=1.10.2rc3
-
-
-# Set AIRFLOW_REPO_ROOT to the path of your git repo
-export AIRFLOW_REPO_ROOT=$(pwd)
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
-# Example after cloning
-git clone https://github.com/apache/airflow.git airflow
-cd airflow
-export AIRFLOW_REPO_ROOT=$(pwd)
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
```
-- set your version to 1.10.2 in airflow/version.py (without the RC tag)
-- Commit the version change.
-
-- Tag your release
+#### Building your own docker image
-`git tag ${VERSION}`
+If you prefer to build your own image, You can also use the official image and PyPI packages to test
+backport packages. If you need to fully test the integration, sometimes you also have to install
+additional components. Below is an example Dockerfile, which installs backport providers for Google
-- Clean the checkout: the sdist step below will
-`git clean -fxd`
+```
+FROM apache/airflow:1.10.12
-- Tarball the repo
-`git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz`
+RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1
-- Generate sdist
-NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.
-`python setup.py compile_assets sdist bdist_wheel`
+RUN curl https://sdk.cloud.google.com | bash \
+ && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
+ && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
-- Rename the sdist
-```
-mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
-mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
-```
+USER 0
+RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
+ && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
+ && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
+ && chmod +x /usr/local/bin/kubectl
-- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
-```
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl
+USER ${AIRFLOW_UID}
```
-- Push Tags
-`git push --tags`
+To build an image build and run a shell, run:
-- Push the artifacts to ASF dev dist repo
```
-# First clone the repo
-svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
-
-# Create new folder for the release
-cd airflow-dev
-svn mkdir ${VERSION}
-
-# Move the artifacts to svn folder & commit
-mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* ${VERSION}/
-cd ${VERSION}
-svn add *
-svn commit -m "Add artifacts for Airflow ${VERSION}"
+docker build . -t my-airflow
+docker run -ti \
+ --rm \
+ -v "$PWD/data:/opt/airflow/" \
+ -v "$PWD/keys/:/keys/" \
+ -p 8080:8080 \
+ -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
+ -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
+ my-airflow
Review comment:
```suggestion
my-airflow bash
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
```Dockerfile
FROM apache/airflow:1.10.12
RUN BACKPORT_RELEASE=2020.10.5rc1 \
&& pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
&& pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
RUN curl https://sdk.cloud.google.com | bash \
&& echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
&& echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
USER 0
RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
&& KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
&& curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
USER ${AIRFLOW_UID}
```
Feel free to modify this example to test your use cases.
To build an image build and run a shell, run:
```bash
docker build . -t my-airflow
docker run -ti \
--rm \
-v "$PWD/data:/opt/airflow/" \
-v "$PWD/keys/:/keys/" \
-p 8080:8080 \
-e AIRFLOW__CORE__LOAD_EXAMPLES=True \
my-airflow
```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
>. -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
##########
File path: dev/README.md
##########
@@ -211,364 +1091,295 @@ Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
```
-# Verifying if the release candidate "works" by Contributors
+### Verify if the Backport Packages release candidates "work" by Contributors
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
is available on PyPI apart from SVN packages, so everyone should be able to install
the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
release candidate number 1,2,3,....).
-```bash
-pip install apache-airflow==<VERSION>rc<X>
-```
-Optionally it can be followed with constraints
-
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
-Note that the constraints contain python version that you are installing it with.
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+#### Installing in your local virtualenv
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
+You have to make sure you have Airilow 1.10.* installed in your PIP virtualenv
+(the version you want to install providers with).
-```
-./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
```
-Once you install and run Airflow, you should perform any verification you see as necessary to check
-that the Airflow works as you expected.
+#### Installing with Breeze
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
-# Building an RC
-
-The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
+First copy all the provider packages .whl files to the `dist` folder.
-- Set environment variables
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
```
-# Set Version
-export VERSION=1.10.2rc3
-
-
-# Set AIRFLOW_REPO_ROOT to the path of your git repo
-export AIRFLOW_REPO_ROOT=$(pwd)
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
-# Example after cloning
-git clone https://github.com/apache/airflow.git airflow
-cd airflow
-export AIRFLOW_REPO_ROOT=$(pwd)
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
```
-- set your version to 1.10.2 in airflow/version.py (without the RC tag)
-- Commit the version change.
-
-- Tag your release
+#### Building your own docker image
-`git tag ${VERSION}`
+If you prefer to build your own image, You can also use the official image and PyPI packages to test
+backport packages. If you need to fully test the integration, sometimes you also have to install
+additional components. Below is an example Dockerfile, which installs backport providers for Google
-- Clean the checkout: the sdist step below will
-`git clean -fxd`
+```
+FROM apache/airflow:1.10.12
-- Tarball the repo
-`git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz`
+RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1
-- Generate sdist
-NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.
-`python setup.py compile_assets sdist bdist_wheel`
+RUN curl https://sdk.cloud.google.com | bash \
+ && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
+ && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
-- Rename the sdist
-```
-mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
-mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
-```
+USER 0
+RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
+ && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
+ && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
+ && chmod +x /usr/local/bin/kubectl
-- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
-```
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
-${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl
+USER ${AIRFLOW_UID}
```
-- Push Tags
-`git push --tags`
+To build an image build and run a shell, run:
-- Push the artifacts to ASF dev dist repo
```
-# First clone the repo
-svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
-
-# Create new folder for the release
-cd airflow-dev
-svn mkdir ${VERSION}
-
-# Move the artifacts to svn folder & commit
-mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* ${VERSION}/
-cd ${VERSION}
-svn add *
-svn commit -m "Add artifacts for Airflow ${VERSION}"
+docker build . -t my-airflow
+docker run -ti \
+ --rm \
+ -v "$PWD/data:/opt/airflow/" \
+ -v "$PWD/keys/:/keys/" \
+ -p 8080:8080 \
+ -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
+ -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
+ my-airflow
Review comment:
```suggestion
my-airflow bash
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-705410793
cc @mucio - your 'start-airfow' command simplified a bit instructions on testing release candidates in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501935259
##########
File path: BREEZE.rst
##########
@@ -2150,6 +2155,11 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
+ --install-wheels
Review comment:
What is the reason to have duplicated information?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501537181
##########
File path: dev/README.md
##########
@@ -388,8 +341,11 @@ with INSTALL instructions.
Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS
-Only votes from PMC members are binding, but members of the community are
-encouraged to test the release and vote with "(non-binding)".
+Only votes from PMC members are binding, but the release manager should encourage members of the community
+to test the release and vote with "(non-binding)".
+
+The test procedure for PMCs and Contributors who would like to test this RC are described in
Review comment:
@ashb @kaxil @mik-laj -> also here, the link to test procedure for PMCs and contributors is added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-706963759
Mistake probably when rebasing (I likely forgot to pull it) Can you make PR restoring it ? I will gladly review it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-704872572
Hey @kaxil @ashb -> I think this change has everything nicely laid out in a bit better structured way where common prerequisites are separated out and release process is documented separately for "Airflow" and "Backport Providers". I am going to update it as well to include the "Providers" separately - and I think next release of the providers can be done by someone else from the PMCs as well.
I think we should have more people capable/familiar with the release process. I also plan to add "docker image" convenience package release (this step is missing for now).
I also tried to capture the important aspects we discussed before - which part of the release process are the official releases, and which are the convenience packages so that a new person doing the release can understand it better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-705717956
All fixed @kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501935259
##########
File path: BREEZE.rst
##########
@@ -2150,6 +2155,11 @@ This is the current syntax for `./breeze <./breeze>`_:
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
+ --install-wheels
Review comment:
What is the reason to have duplicated information?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-705408793
@kaxil @ash - this one now reflects the reality - i updated all the tools and went through backport release procedure according to this one and also added explanation about how to start and run airflow 1.10 and backports for PMC/Contributor's testing (recently added --install-wheels --no-rbac-ui flags).
This will also be super useful for 2.0 when we split the providers. Looking forward for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-706769769
@kaxil @ashb @mik-laj - > would love an approval here as the doc-only change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501937645
##########
File path: dev/README.md
##########
@@ -567,8 +704,645 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
Review comment:
I propose to add the example below as some people like to have more control over their environment. This will allow us to better test the new versions in different edge cases and configurations.
> You can also use the official image and PyPI packages to test backport packages. If you need to fully test the integration, sometimes you also have to install additional components. Below is Dockerfile, which installs `gcloud`, `kubectl` and backport providers for Google and Kubernetes.
>
> ```Dockerfile
> FROM apache/airflow:1.10.12
>
> RUN BACKPORT_RELEASE=2020.10.5rc1 \
> && pip install --user "apache-airflow-backport-providers-google==${BACKPORT_RELEASE}" \
> && pip install --user pip "install apache-airflow-backport-providers-cncf-kubernetes==${BACKPORT_RELEASE}"
>
> RUN curl https://sdk.cloud.google.com | bash \
> && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
> && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
>
> USER 0
> RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
> && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
> && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
> && chmod +x /usr/local/bin/kubectl
>
> USER ${AIRFLOW_UID}
> ```
> Feel free to modify this example to test your use cases.
>
> To build an image build and run a shell, run:
> ```bash
> docker build . -t my-airflow
> docker run -ti \
> --rm \
> -v "$PWD/data:/opt/airflow/" \
> -v "$PWD/keys/:/keys/" \
> -p 8080:8080 \
> -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
> -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
> my-airflow
> ```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-706962530
@potiuk Is there any reason why the Dockerfile test guides was not included in the final version of this change?
https://github.com/apache/airflow/pull/11310#discussion_r501937645
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#issuecomment-706370099
[The Workflow run](https://github.com/apache/airflow/actions/runs/298120781) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r500938468
##########
File path: dev/README.md
##########
@@ -575,7 +703,641 @@ https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
Cheers,
<your name>
-</p>
-</details>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Backport Provider Packages
+
+You can read more about the command line tools used to generate backport packages in
+[Backport Providers README](../backport_packages/README.md).
+
+## Decide when to release
+
+You can release backport packages separately on an ad-hoc basis, whenever we find that a given provider needs
+to be released - due to new features or due to bug fixes. You can release each backport package
+separately.
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately.
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```
+./breeze prepare-backport-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```
+./breeze prepare-backport-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```
+./backport_packages/build_source_package.sh
+
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
+
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
+
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+TODO: Paste the result of twine upload
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You have to make sure you have ariflow 1.10.* (the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --instal-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
+Once you install and run Airflow, you should perform any verification you see as necessary to check
+that the Airflow works as you expected.
+
+
+## Publish the final releases of backport packages
+
+### Summarize the voting for the Backport Providers Release
+
+Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:
+
+Subject:
+```shell script
+cat <<EOF
+[RESULT][VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+Body:
+
+```shell script
+cat <<EOF
+
+Hey all,
+
+Airflow Backport Providers ${VERSION} (based on the ${VERSION_RC} candidate) has been accepted.
+
+N "+1" binding votes received:
+- PMC Member (binding)
+...
+
+N "+1" non-binding votes received:
+
+- COMMITER (non-binding)
+
+Vote thread:
+https://lists.apache.org/thread.html/<TODO:REPLACE_ME_WITH_THE_VOTING_THREAD>@%3Cdev.airflow.apache.org%3E
+
+I'll continue with the release process and the release announcement will follow shortly.
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+
+```
+
+### Publish release to SVN
+
+The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries
+again, and gives a clearer history in the svn commit logs.
+
+We also need to archive older releases before copying the new ones
+[Release policy](http://www.apache.org/legal/release-policy.html#when-to-archive)
+
+```shell script
+# Set the variables
+export VERSION_RC=2020.5.20rc2
+export VERSION=${VERSION_RC/rc?/}
+
+# Set AIRFLOW_REPO_ROOT to the path of your git repo
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+# Go to the directory where you have checked out the dev svn release
+# And go to the sub-folder with RC candidates
+cd "<ROOT_OF_YOUR_DEV_REPO>/backport-providers/${VERSION_RC}"
+export SOURCE_DIR=$(pwd)
+
+# Go the folder where you have checked out the release repo
+# Clone it if it's not done yet
+svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
+
+# Update to latest version
+svn update
+
+# Create backport-providers folder if it does not exist
+# All latest releases are kept in this one folder without version sub-folder
+mkdir -pv backport-providers
+cd backport-providers
+
+# Move the artifacts to svn folder & remove the rc postfix
+for file in ${SOURCE_DIR}/*${VERSION_RC}*
+do
+ base_file=$(basename ${file})
+ svn cp "${file}" "${base_file/${VERSION_RC}/${VERSION}}"
+done
+
+
+# If some packages have been excluded, remove them now
+# Check the packages
+ls *<provider>*
+# Remove them
+svn rm *<provider>*
+
+# Check which old packages will be removed (you need python 3.6+)
+python ${AIRFLOW_REPO_ROOT}/backport_packages/remove_old_releases.py \
+ --directory .
+
+# Remove those packages
+python ${AIRFLOW_REPO_ROOT}/backport_packages/remove_old_releases.py \
+ --directory . --execute
+
+
+# Commit to SVN
+svn commit -m "Release Airflow Backport Providers ${VERSION} from ${VERSION_RC}"
+```
+
+Verify that the packages appear in
+[backport-providers](https://dist.apache.org/repos/dist/release/airflow/backport-providers)
+
+### Publish the final version convenience package to PyPI
+
+Checkout the RC Version:
+
+```shell script
+git checkout backport-providers-${VERSION_RC}
+```
+
+Tag and push the final version (providing that your apache remote is named 'apache'):
+
+```shell script
+git tag backport-providers-${VERSION}
+git push apache backport-providers-${VERSION}
+```
+
+In order to publish to PyPI you just need to build and release packages.
+
+* Generate the packages.
+
+```shell script
+./breeze prepare-backport-packages
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze prepare-backport-packages
+```
+
+In case you decided to remove some of the packages. Remove them from dist folder now:
+
+```shell script
+ls dist/*<provider>*
+rm dist/*<provider>*
+```
+
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+### Notify developers of release
+
+- Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that
+the artifacts have been published:
+
+### Notify developers of release
+
+- Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that
+the artifacts have been published:
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11310: Merge Airflow and Backport Packages preparation instructions
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11310:
URL: https://github.com/apache/airflow/pull/11310#discussion_r501603182
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
-* -bin-tar.gz + .asc + .sha512
-* -source.tar.gz + .asc + .sha512
-* -.whl + .asc + .sha512
+There are also 50+ convenience packages released as "apache-airflow-backport-providers" separately in
+PyPI. You can find them all by [PyPI query](https://pypi.org/search/?q=apache-airflow-backport-providers)
-As a PMC you should be able to clone the SVN repository:
+The document describes the process of releasing both - official source packages and convenience
+packages for Backport Provider Packages.
-```bash
-svn co https://dist.apache.org/repos/dist/dev/airflow
-```
+# Prerequisites for the release manager preparing the release
-Or update it if you already checked it out:
+The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
+available in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the release manager
+can ask PMC to perform that final step of release.
-```bash
-svn update .
-```
+## Upload Public keys to id.apache.org
-## Verifying the licences
+Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
-This can be done with the Apache RAT tool.
+- Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of that file. Upload your GPG public key to https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache credentials, paste your fingerprint into the pgp fingerprint field and hit save).
-* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
- the jar is inside)
-* Unpack the -source.tar.gz to a folder
-* Enter the folder and run the check (point to the place where you extracted the .jar)
+```shell script
+# Create PGP Key
+gpg --gen-key
-```bash
-java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
-```
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/airflow
+cd airflow
-## Verifying the signatures
-Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
+(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS
-You can import the whole KEYS file:
-```bash
-gpg --import KEYS
+# Commit the changes
+svn commit -m "Add PGP keys of Airflow developers"
```
-You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
-retrieves it from the default GPG keyserver
-[OpenPGP.org](https://keys.openpgp.org):
-
-```bash
-gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+See this for more detail on creating keys and what is required for signing releases.
-You should choose to import the key when asked.
+http://www.apache.org/dev/release-signing.html#basic-facts
-Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
-errors or timeouts. Many of the release managers also uploaded their keys to the
-[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+## Configure PyPI uploads
-```bash
-gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens.
+You can add and copy the tokens here:
-Once you have the keys, the signatures can be verified by running this:
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
-```bash
-for i in *.asc
-do
- echo "Checking $i"; gpg --verify `basename $i .sha512 `
-done
-```
-This should produce results similar to the below. The "Good signature from ..." is indication
-that the signatures are correct. Do not worry about the "not certified with a trusted signature"
-warning. Most of certificates used by release managers are self signed, that's why you get this
-warning. By importing the server in the previous step and importing it via ID from
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
-this is a valid Key already.
+Create a ~/.pypirc file:
-```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
-gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
-gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-```
+```shell script
+[distutils]
+index-servers =
+ pypi
+ pypitest
-## Verifying the SHA512 sum
+[pypi]
+username=__token__
+password=<API Upload Token>
-Run this:
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
-```bash
-for i in *.sha512
-do
- echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
-done
```
-You should get output similar to:
+Set proper permissions for the pypirc file:
+```shell script
+$ chmod 600 ~/.pypirc
```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
-Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
-```
-
-# Verifying if the release candidate "works" by Contributors
-This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
-actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
-is available on PyPI apart from SVN packages, so everyone should be able to install
-the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
-release candidate number 1,2,3,....).
+* Install twine if you do not have it already.
-```bash
-pip install apache-airflow==<VERSION>rc<X>
+```shell script
+pip install twine
```
-Optionally it can be followed with constraints
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
-Note that the constraints contain python version that you are installing it with.
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+- Confirm that airflow/version.py is set properly.
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Here is a typical scenario:
+- Install [twine](https://pypi.org/project/twine/) if you do not have it already (it can be done
+ in a separate virtual environment).
+ `pip install twine`
Review comment:
Duplicate of Line 185
##########
File path: dev/README.md
##########
@@ -20,240 +20,196 @@
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
-- [Development Tools](#development-tools)
- - [Airflow release signing tool](#airflow-release-signing-tool)
-- [Verifying the release candidate by PMCs (legal)](#verifying-the-release-candidate-by-pmcs-legal)
- - [PMC voting](#pmc-voting)
- - [SVN check](#svn-check)
- - [Verifying the licences](#verifying-the-licences)
- - [Verifying the signatures](#verifying-the-signatures)
- - [Verifying the SHA512 sum](#verifying-the-sha512-sum)
-- [Verifying if the release candidate "works" by Contributors](#verifying-if-the-release-candidate-works-by-contributors)
-- [Building an RC](#building-an-rc)
-- [PyPI Snapshots](#pypi-snapshots)
-- [Make sure your public key is on id.apache.org and in KEYS](#make-sure-your-public-key-is-on-idapacheorg-and-in-keys)
-- [Voting on an RC](#voting-on-an-rc)
-- [Publishing release](#publishing-release)
-- [Publishing to PyPi](#publishing-to-pypi)
-- [Updating CHANGELOG.md](#updating-changelogmd)
-- [Notifying developers of release](#notifying-developers-of-release)
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages.](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Backport Provider Packages](#backport-provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-# Development Tools
+# Apache Airflow source releases
-## Airflow release signing tool
+The Apache Airflow releases are one of the two types:
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
-### Execution
+## Apache Airflow Package
-To create a release tarball execute following command from Airflow's root.
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
-```bash
-python setup.py compile_assets sdist --formats=gztar
-```
-
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
-After that navigate to relative directory i.e., `cd dist` and sign the release files.
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
-```bash
-../dev/sign.sh <the_created_tar_ball.tar.gz
-```
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
-Signing files will be created in the same directory.
+Following source releases Apache Airflow release manager also distributes convenience packages:
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
-# Verifying the release candidate by PMCs (legal)
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
-## PMC voting
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
-The PMCs should verify the releases in order to make sure the release is following the
-[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+## Backport Provider packages
-At least 3 (+1) votes should be recorded in accordance to
-[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
-The legal checks include:
+Once you release the packages, you can simply install them with:
-* checking if the packages are present in the right dist folder on svn
-* verifying if all the sources have correct licences
-* verifying if release manager signed the releases with the right key
-* verifying if all the checksums are valid for the release
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
-## SVN check
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
-The files should be present in the sub-folder of
-[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
-The following files should be present (9 files):
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources]https://downloads.apache.org/airflow/backport-providers/.
-* -bin-tar.gz + .asc + .sha512
-* -source.tar.gz + .asc + .sha512
-* -.whl + .asc + .sha512
+There are also 50+ convenience packages released as "apache-airflow-backport-providers" separately in
+PyPI. You can find them all by [PyPI query](https://pypi.org/search/?q=apache-airflow-backport-providers)
-As a PMC you should be able to clone the SVN repository:
+The document describes the process of releasing both - official source packages and convenience
+packages for Backport Provider Packages.
-```bash
-svn co https://dist.apache.org/repos/dist/dev/airflow
-```
+# Prerequisites for the release manager preparing the release
-Or update it if you already checked it out:
+The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
+available in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the release manager
+can ask PMC to perform that final step of release.
-```bash
-svn update .
-```
+## Upload Public keys to id.apache.org
-## Verifying the licences
+Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
-This can be done with the Apache RAT tool.
+- Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of that file. Upload your GPG public key to https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache credentials, paste your fingerprint into the pgp fingerprint field and hit save).
-* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
- the jar is inside)
-* Unpack the -source.tar.gz to a folder
-* Enter the folder and run the check (point to the place where you extracted the .jar)
+```shell script
+# Create PGP Key
+gpg --gen-key
-```bash
-java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
-```
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/airflow
+cd airflow
-## Verifying the signatures
-Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
+(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS
-You can import the whole KEYS file:
-```bash
-gpg --import KEYS
+# Commit the changes
+svn commit -m "Add PGP keys of Airflow developers"
```
-You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
-retrieves it from the default GPG keyserver
-[OpenPGP.org](https://keys.openpgp.org):
-
-```bash
-gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+See this for more detail on creating keys and what is required for signing releases.
-You should choose to import the key when asked.
+http://www.apache.org/dev/release-signing.html#basic-facts
-Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
-errors or timeouts. Many of the release managers also uploaded their keys to the
-[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+## Configure PyPI uploads
-```bash
-gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-```
+In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens.
+You can add and copy the tokens here:
-Once you have the keys, the signatures can be verified by running this:
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
-```bash
-for i in *.asc
-do
- echo "Checking $i"; gpg --verify `basename $i .sha512 `
-done
-```
-This should produce results similar to the below. The "Good signature from ..." is indication
-that the signatures are correct. Do not worry about the "not certified with a trusted signature"
-warning. Most of certificates used by release managers are self signed, that's why you get this
-warning. By importing the server in the previous step and importing it via ID from
-[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
-this is a valid Key already.
+Create a ~/.pypirc file:
-```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
-gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
-gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
-gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
-gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
-gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
-gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
-gpg: WARNING: This key is not certified with a trusted signature!
-gpg: There is no indication that the signature belongs to the owner.
-Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
-```
+```shell script
+[distutils]
+index-servers =
+ pypi
+ pypitest
-## Verifying the SHA512 sum
+[pypi]
+username=__token__
+password=<API Upload Token>
-Run this:
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
-```bash
-for i in *.sha512
-do
- echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
-done
```
-You should get output similar to:
+Set proper permissions for the pypirc file:
+```shell script
+$ chmod 600 ~/.pypirc
```
-Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
-Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
-Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
-```
-
-# Verifying if the release candidate "works" by Contributors
-This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
-actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
-is available on PyPI apart from SVN packages, so everyone should be able to install
-the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
-release candidate number 1,2,3,....).
+* Install twine if you do not have it already.
-```bash
-pip install apache-airflow==<VERSION>rc<X>
+```shell script
+pip install twine
```
-Optionally it can be followed with constraints
-```bash
-pip install apache-airflow==<VERSION>rc<X> \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
-```
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
-Note that the constraints contain python version that you are installing it with.
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
-You can use any of the installation methods you prefer (you can even install it via the binary wheel
-downloaded from the SVN).
+- Confirm that airflow/version.py is set properly.
-There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
-Here is a typical scenario:
+- Install [twine](https://pypi.org/project/twine/) if you do not have it already (it can be done
+ in a separate virtual environment).
+ `pip install twine`
-1. `./breeze --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres`
-2. `tmux`
-3. Hit Ctrl-B followed by "
-4. `airflow resetdb -y`
-5. if you want RBAC:
- * Change RBAC setting: `sed "s/rbac = False/rbac = True/" -i /root/airflow/airflow.cfg`
- * airflow resetdb -y
- * Run`airflow create_user -r Admin -u airflow -e airflow@apache.org -f Airflow -l User -p airflow
-6. `airflow scheduler`
-7. Ctrl-B "up-arrow"
-8. `airflow webserver`
+## Hardware used to prepare and verify the packages.
Review comment:
```suggestion
## Hardware used to prepare and verify the packages
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org