You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/12/30 12:20:18 UTC
(airflow) 15/27: Update release process for providers to include mixed RC versions (#36385)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e49c573d9ce03066c61cce0624576700de1de07a
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Dec 23 17:46:29 2023 +0100
Update release process for providers to include mixed RC versions (#36385)
This PR updates released process for providers to enable releasing
providers in more regular batches. Sometimes when we exclude a
provider from previous voting, we want to release RCN (2,3 etc.)
candidate.
However, especially when time between previous RC and the new one
is long (for example because fixing took a long time) we might
want to release the RCN release for that cancelled providers and
RC1 for all the providers that have been changed in the meantime.
This cchange makes it possible (and easy):
1) release RC1 for all providers (the RCN provider should be skipped,
because tag for this provider already exists.
2) release the RCN providers with `--version-suffix-for-pypi rcN`.
The release process and tools were updated to account for that - where
rc candidate number is retrieved from packages prepared in `dist`.
Fixed a few small missing things in the process.
(cherry picked from commit 4deed641322343cb18eabd9ad199f5ac83f29ff0)
---
dev/README_RELEASE_PROVIDER_PACKAGES.md | 20 ++++++++++++--------
.../commands/release_management_commands.py | 22 +++++++++++++++++++++-
.../provider_issue_TEMPLATE.md.jinja2 | 2 +-
3 files changed, 34 insertions(+), 10 deletions(-)
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index f281b8403f..2c7830a413 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -330,13 +330,21 @@ breeze release-management prepare-provider-packages --include-removed-providers
--version-suffix-for-pypi rc1 --package-format both
```
-if you only build few packages, run:
+If you only build few packages, run:
```shell script
breeze release-management prepare-provider-packages \
--version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
```
+In case you are ALSO releasing RC2, RC3, etc. for selected packages, they will be skipped automatically because
+the `rc1` tag will be created for them already. In this case you should specify the ``rc*`` that you want to
+build and specify the package id's you want to build.
+
+```shell script
+breeze release-management prepare-provider-packages --version-suffix-for-pypi rc2 --package-format both PACKAGE
+```
+
* Verify the artifacts that would be uploaded:
```shell script
@@ -416,7 +424,7 @@ git pull --rebase
```shell script
cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build apache-airflow-providers all-providers
+breeze build-docs --clean-build apache-airflow-providers all-providers --include-removed-providers
```
Usually when we release packages we also build documentation for the "documentation-only" packages. This
@@ -461,12 +469,6 @@ breeze release-management publish-docs apache-airflow-providers all-providers --
breeze release-management add-back-references all-providers
```
-If you see `ModuleNotFoundError: No module named 'docs'`, set:
-
-```
-export PYTHONPATH=.:${PYTHONPATH}
-```
-
If you have providers as list of provider ids because you just released them you can build them with
```shell script
@@ -503,6 +505,8 @@ You can also pass the token as `--github-token` option in the script.
You can also pass list of PR to be excluded from the issue with `--excluded-pr-list`.
```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+
breeze release-management generate-issue-content-providers --only-available-in-dist
```
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 4a6a2e996a..fba5855ef1 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -1495,6 +1495,23 @@ def is_package_in_dist(dist_files: list[str], package: str) -> bool:
)
+VERSION_MATCH = re.compile(r"([0-9]+)\.([0-9]+)\.([0-9]+)(.*)")
+
+
+def get_suffix_from_package_in_dist(dist_files: list[str], package: str) -> str | None:
+ """Get suffix from package prepared in dist folder."""
+ for file in dist_files:
+ if file.startswith(f'apache_airflow_providers_{package.replace(".", "_")}') and file.endswith(
+ ".tar.gz"
+ ):
+ file = file[: -len(".tar.gz")]
+ version = file.split("-")[-1]
+ match = VERSION_MATCH.match(version)
+ if match:
+ return match.group(4)
+ return None
+
+
def get_prs_for_package(provider_id: str) -> list[int]:
pr_matcher = re.compile(r".*\(#([0-9]*)\)``$")
prs = []
@@ -1566,6 +1583,7 @@ def generate_issue_content_providers(
pypi_package_name: str
version: str
pr_list: list[PullRequest.PullRequest | Issue.Issue]
+ suffix: str
if not provider_packages:
provider_packages = list(DEPENDENCIES.keys())
@@ -1622,16 +1640,18 @@ def generate_issue_content_providers(
).read_text()
)
if pull_request_list:
+ package_suffix = get_suffix_from_package_in_dist(files_in_dist, provider_id)
providers[provider_id] = ProviderPRInfo(
version=provider_yaml_dict["versions"][0],
provider_package_id=provider_id,
pypi_package_name=provider_yaml_dict["package-name"],
pr_list=pull_request_list,
+ suffix=package_suffix if package_suffix else suffix,
)
template = jinja2.Template(
(Path(__file__).parents[1] / "provider_issue_TEMPLATE.md.jinja2").read_text()
)
- issue_content = template.render(providers=providers, date=datetime.now(), suffix=suffix)
+ issue_content = template.render(providers=providers, date=datetime.now())
get_console().print()
get_console().print(
"[green]Below you can find the issue content that you can use "
diff --git a/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2 b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
index 1639a8857d..8009549772 100644
--- a/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
+++ b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
@@ -10,7 +10,7 @@ Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
{% for provider_id, provider_info in providers.items() %}
-## Provider [{{ provider_id }}: {{ provider_info.version }}{{ suffix }}](https://pypi.org/project/{{ provider_info.pypi_package_name }}/{{ provider_info.version }}{{ suffix }})
+## Provider [{{ provider_id }}: {{ provider_info.version }}{{ provider_info.suffix }}](https://pypi.org/project/{{ provider_info.pypi_package_name }}/{{ provider_info.version }}{{ provider_info.suffix }})
{%- for pr in provider_info.pr_list %}
- [ ] [{{ pr.title }} (#{{ pr.number }})]({{ pr.html_url }}): @{{ pr.user.login }}
{%- endfor %}