[airflow] branch master updated: Updates provider release process (#15339)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
     new 7490c6b  Updates provider release process (#15339)
7490c6b is described below

commit 7490c6b8109adcc5aec517fc8d39cfc31912d92f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Apr 13 22:44:34 2021 +0200

    Updates provider release process (#15339)
    
    Updates the provider release process with approach where
    Final PyPI upload are the same as SVN uploads.
    
    This way we avoid re-building packages again when we release PyPI
    version and the files which get uploaded to PyPI are the same that
    are stored in SVN.
---
 dev/README_RELEASE_PROVIDER_PACKAGES.md            | 54 +++++++++-------------
 dev/provider_packages/prepare_provider_packages.py |  5 +-
 2 files changed, 27 insertions(+), 32 deletions(-)

diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index 3eb3871..1c5ba8a 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -158,6 +158,9 @@ svn update
 # Create a new folder for the release.
 cd providers
 
+# Remove previously released providers
+rm -rf *
+
 # Move the artifacts to svn folder
 mv ${AIRFLOW_REPO_ROOT}/dist/* .
 
@@ -652,14 +655,21 @@ We also need to archive older releases before copying the new ones
 [Release policy](http://www.apache.org/legal/release-policy.html#when-to-archive)
 
 ```shell script
+# Go to the directory where you have checked out the dev svn release
+# And go to the sub-folder with RC candidates
+cd "<ROOT_OF_YOUR_AIRFLOW_REPO>"
 # Set AIRFLOW_REPO_ROOT to the path of your git repo
 export AIRFLOW_REPO_ROOT=$(pwd)
 
-# Go to the directory where you have checked out the dev svn release
-# And go to the sub-folder with RC candidates
 cd "<ROOT_OF_YOUR_DEV_REPO>/providers/"
 export SOURCE_DIR=$(pwd)
 
+# If some packages have been excluded, remove them now
+# Check the packages
+ls *<provider>*
+# Remove them
+svn rm *<provider>*
+
 # Go the folder where you have checked out the release repo
 # Clone it if it's not done yet
 svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
@@ -672,20 +682,16 @@ svn update
 mkdir -pv providers
 cd providers
 
-# Move the artifacts to svn folder & remove the rc postfix
+# Copy your providers with the target name to dist directory and to SVN
+rm ${AIRFLOW_REPO_ROOT}/dist/*
+
 for file in ${SOURCE_DIR}/*
 do
  base_file=$(basename ${file})
- svn mv "${file}" "${base_file//rc[0-9][\.-]/.}"
+ cp -v "${file}" "${AIRFLOW_REPO_ROOT}/dist/${base_file//rc[0-9]/}"
+ svn mv "${file}" "${base_file//rc[0-9]/}"
 done
 
-
-# If some packages have been excluded, remove them now
-# Check the packages
-ls *<provider>*
-# Remove them
-svn rm *<provider>*
-
 # Check which old packages will be removed (you need python 3.6+)
 python ${AIRFLOW_REPO_ROOT}/dev/provider_packages/remove_old_releases.py \
     --directory .
@@ -705,37 +711,23 @@ Verify that the packages appear in
 
 ## Publish the Regular convenience package to PyPI
 
-* Checkout the RC Version for the RC Version released (there is a batch of providers - one of them is enough):
-
-    ```shell script
-    git checkout providers-<PROVIDER_NAME>/<VERSION_RC>
-    ```
-
-* Generate the packages with final version. Note that
-  this will clean up dist folder before generating the packages, so you will only have the right packages there.
-
-```shell script
-rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
-./breeze prepare-provider-packages --package-format both
-```
-
-if you ony build few packages, run:
+By that time the packages with proper name (renamed from rc* to final version should be in your dist
+folder.
 
 ```shell script
-rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
-./breeze prepare-provider-packages --package-format both PACKAGE PACKAGE ....
+cd ${AIRFLOW_REPO_ROOT}
 ```
 
 * Verify the artifacts that would be uploaded:
 
 ```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*
+twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
 ```
 
 * Upload the package to PyPi's test environment:
 
 ```shell script
-twine upload -r pypitest ${AIRFLOW_REPO_ROOT}/dist/*
+twine upload -r pypitest ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
 ```
 
 * Verify that the test packages look good by downloading it and installing them into a virtual environment.
@@ -744,7 +736,7 @@ twine upload -r pypitest ${AIRFLOW_REPO_ROOT}/dist/*
 * Upload the package to PyPi's production environment:
 
 ```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
+twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
 ```
 
 * Again, confirm that the packages are available under the links printed.
diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
index dd8e13f..81bff99 100755
--- a/dev/provider_packages/prepare_provider_packages.py
+++ b/dev/provider_packages/prepare_provider_packages.py
@@ -1815,10 +1815,13 @@ def get_current_tag(provider_package_id: str, suffix: str, git_update: bool, ver
 
 def cleanup_remnants(verbose: bool):
     if verbose:
-        print("Cleaning remnants (*.egginfo)")
+        print("Cleaning remnants")
     files = glob.glob("*.egg-info")
     for file in files:
         shutil.rmtree(file, ignore_errors=True)
+    files = glob.glob("build")
+    for file in files:
+        shutil.rmtree(file, ignore_errors=True)
 
 
 def verify_setup_py_prepared(provider_package):