You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ms...@apache.org on 2021/04/12 11:04:41 UTC
[airflow-client-python] branch master updated: Add
README_RELEASE_CLIENT (#10)
This is an automated email from the ASF dual-hosted git repository.
msumit pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow-client-python.git
The following commit(s) were added to refs/heads/master by this push:
new dd1b4ad Add README_RELEASE_CLIENT (#10)
dd1b4ad is described below
commit dd1b4adc261bbd99fb13804cdf30beb20d323eef
Author: Sumit Maheshwari <ms...@users.noreply.github.com>
AuthorDate: Mon Apr 12 16:34:33 2021 +0530
Add README_RELEASE_CLIENT (#10)
---
.pre-commit-config.yaml | 4 +-
dev/README_RELEASE_CLIENT.md | 269 +++++++++++++++++++++++++++++++++++++++++++
dev/sign.sh | 34 ++++++
requirements.txt | 29 ++---
test-requirements.txt | 29 ++---
5 files changed, 335 insertions(+), 30 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index ea9d354..afda3cc 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -58,7 +58,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all other files
- exclude: ^\.github/.*$
+ exclude: ^\.github/.*$|\.*LICENSE.txt$
args:
- --comment-style
- "|#|"
@@ -66,4 +66,4 @@ repos:
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
files: >
- \.properties$|\.cfg$|\.conf$|\.ini$$
\ No newline at end of file
+ \.properties$|\.cfg$|\.conf$|\.ini$|\.txt$$
\ No newline at end of file
diff --git a/dev/README_RELEASE_CLIENT.md b/dev/README_RELEASE_CLIENT.md
new file mode 100644
index 0000000..90d260e
--- /dev/null
+++ b/dev/README_RELEASE_CLIENT.md
@@ -0,0 +1,269 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+You can find the prerequisites to release Apache Airflow Python Client in [README.md](README.md).
+
+# Prepare the Apache Airflow Python Client Package RC
+
+## Build RC artifacts
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than
+renaming – i.e. the contents of the files must be the same between voted release candidate and final release. Because
+of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
+
+- Set environment variables
+
+ ```shell script
+ # Set Version
+ export VERSION=2.0.0rc1
+
+
+ # Example after cloning
+ git clone https://github.com/apache/airflow-client-python.git airflow-client
+ cd airflow-client
+ export CLIENT_REPO_ROOT=$(pwd)
+ ```
+
+- Set your version to 2.0.0 in `setup.py` (without the RC tag)
+- Commit the version change.
+
+- Tag your release
+
+ ```shell script
+ git tag -s ${VERSION}
+ ```
+
+- Clean the checkout: the sdist step below will
+
+ ```shell script
+ git clean -fxd
+ ```
+
+- Tarball the repo
+
+ ```shell script
+ git archive --format=tar.gz ${VERSION} -o airflow-client-${VERSION}-source.tar.gz
+ ```
+
+- Generate sdist
+
+ NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included
+ in the file produced.
+
+ ```shell script
+ python setup.py sdist bdist_wheel
+ ```
+
+- Rename the sdist
+
+ ```shell script
+ mv dist/airflow-client-${VERSION%rc?}.tar.gz airflow-client-${VERSION}-bin.tar.gz
+ mv dist/airflow_client-${VERSION%rc?}-py3-none-any.whl airflow_client-${VERSION}-py3-none-any.whl
+ ```
+
+- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on
+ http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
+
+ ```shell script
+ ${CLIENT_REPO_ROOT}/dev/sign.sh airflow-client-${VERSION}-source.tar.gz
+ ${CLIENT_REPO_ROOT}/dev/sign.sh airflow-client-${VERSION}-bin.tar.gz
+ ${CLIENT_REPO_ROOT}/dev/sign.sh airflow_client-${VERSION}-py3-none-any.whl
+ ```
+
+- Push the artifacts to ASF dev dist repo
+
+```
+# First clone the repo
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# Create new folder for the release
+cd airflow-dev/clients/python
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder & commit
+mv ${CLIENT_REPO_ROOT}/airflow{-,_}client-${VERSION}* ${VERSION}/
+cd ${VERSION}
+svn add *
+svn commit -m "Add artifacts for Airflow Python Client ${VERSION}"
+```
+
+## Prepare PyPI convenience "snapshot" packages
+
+At this point we have the artefact that we vote on, but as a convenience to developers we also want to
+publish "snapshots" of the RC builds to pypi for installing via pip.
+
+To do this we need to
+
+- Build the package:
+
+ ```shell script
+ cd ${CLIENT_REPO_ROOT}
+ python setup.py egg_info --tag-build "$(sed -e "s/^[0-9.]*//" <<<"$VERSION")" sdist bdist_wheel
+ ```
+
+- Verify the artifacts that would be uploaded:
+
+ ```shell script
+ twine check dist/*
+ ```
+
+- Upload the package to PyPi's test environment:
+
+ ```shell script
+ twine upload --repository-url=https://test.pypi.org/legacy/ dist/*
+ ```
+
+- Verify that the test package looks good by downloading it and installing it into a virtual environment. The package
+ download link is available at:
+ https://test.pypi.org/project/airflow-client/#files
+
+ Or via pypi
+ pip install -i https://test.pypi.org/simple/ airflow-client==$(VERSION)
+
+- Upload the package to PyPi's production environment:
+ `twine upload -r pypi dist/*`
+
+- Again, confirm that the package is available here:
+ https://pypi.python.org/pypi/airflow-client
+
+It is important to stress that this snapshot should not be named "release", and it
+is not supposed to be used by and advertised to the end-users who do not read the devlist.
+
+- Push Tag for the release candidate
+
+ ```shell script
+ git push origin ${VERSION}
+ ```
+
+## Prepare Vote email on the Airflow Client release candidate
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#prepare-vote-email-on-the-apache-airflow-release-candidate) (just replace Airflow with Airflow Client).
+
+# Verify the release candidate by PMCs
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmcs).
+
+## SVN check
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#svn-check) (just replace Airflow with Airflow Client).
+
+## Licence check
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#licence-check).
+
+## Signature check
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#signature-check).
+
+
+
+# Verify release candidates by Contributors
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Airflow Client test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow Client via simply (<VERSION> is 2.0.0 for example, and <X> is
+release candidate number 1,2,3,....).
+
+```shell script
+pip install airflow-client==<VERSION>rc<X>
+```
+
+Once you install and run Airflow Client, you should perform any verification you see as necessary to check
+that the client works as you expected.
+
+# Publish the final Airflow client release
+
+## Summarize the voting for the Airflow client release
+
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#publish-the-final-apache-airflow-release) (just replace Airflow with Airflow Client).
+
+## Publish release to SVN
+
+```shell script
+# First clone the repo
+export RC=2.0.0rc1
+export VERSION=${RC/rc?/}
+svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
+
+# Create new folder for the release
+cd airflow-release/clients/python
+svn mkdir ${VERSION}
+cd ${VERSION}
+
+# Move the artifacts to svn folder & commit
+for f in ../../../airflow-dev/clients/python/$RC/*; do svn cp $f ${$(basename $f)/rc?/}; done
+svn commit -m "Release Airflow Client ${VERSION} from ${RC}"
+
+# Remove old release
+# http://www.apache.org/legal/release-policy.html#when-to-archive
+cd ..
+export PREVIOUS_VERSION=1.0.0
+svn rm ${PREVIOUS_VERSION}
+svn commit -m "Remove old release: ${PREVIOUS_VERSION}"
+```
+
+Verify that the packages appear in [airflow](https://dist.apache.org/repos/dist/release/airflow/clients/python)
+
+## Prepare PyPI "release" packages
+
+At this point we release an official package:
+
+- Build the package:
+
+ ```shell script
+ python setup.py sdist bdist_wheel
+ ```
+
+- Verify the artifacts that would be uploaded:
+
+ ```shell script
+ twine check dist/*
+ ```
+
+- Upload the package to PyPi's test environment:
+
+ ```shell script
+ twine upload --repository-url=https://test.pypi.org/legacy/ dist/*
+ ```
+
+- Verify that the test package looks good by downloading it and installing it into a virtual environment.
+ The package download link is available at: https://test.pypi.org/project/airflow-client/#files
+
+- Upload the package to PyPi's production environment:
+
+ ```shell script
+ twine upload -r pypi dist/*
+ ```
+
+- Again, confirm that the package is available here: https://pypi.python.org/pypi/airflow-client
+
+## Update CHANGELOG.md
+
+- Get a diff between the last version and the current version:
+
+ ```shell script
+ git log 1.0.0..2.0.0 --pretty=oneline
+ ```
+
+- Update CHANGELOG.md with the details, and commit it.
+
+- Push Tag for the final version
+
+ ```shell script
+ git push origin ${VERSION}
+ ```
+
+## Notify developers of release
+
+See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#notify-developers-of-release) (just replace Airflow with Airflow Client)
\ No newline at end of file
diff --git a/dev/sign.sh b/dev/sign.sh
new file mode 100755
index 0000000..a1b6916
--- /dev/null
+++ b/dev/sign.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+# Use this to sign the tar balls generated from
+# python setup.py sdist --formats=gztar
+# ie. sign.sh <my_tar_ball>
+# you will still be required to type in your signing key password
+# or it needs to be available in your keychain
+
+# Which key to sign releases with? This can be a (partial) email address or a
+# key id. By default use any apache.org key
+SIGN_WITH="${SIGN_WITH:-apache.org}"
+
+for name in "${@}"
+do
+ gpg --armor --local-user "$SIGN_WITH" --output "${name}.asc" --detach-sig "${name}"
+ shasum -a 512 "${name}" > "${name}.sha512"
+done
diff --git a/requirements.txt b/requirements.txt
index ce3b2f4..51a300c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,19 +1,20 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
python_dateutil >= 2.5.3
setuptools >= 21.0.0
urllib3 >= 1.25.3
diff --git a/test-requirements.txt b/test-requirements.txt
index 8f2be00..2ccbc58 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -1,19 +1,20 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
pytest-cov>=2.8.1
python_dateutil >= 2.5.3
urllib3 >= 1.25.3