You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/01/12 02:41:40 UTC
[PR] Update Airflow release process to include reproducible tarballs [airflow]
potiuk opened a new pull request, #36744:
URL: https://github.com/apache/airflow/pull/36744
Source tarball is the main artifact produced by the release process - one that is the "official" release and named like that by the Apache Software Foundation.
This PR makes the source tarball generation reproducible - following reproducibility of the `.whl` and `sdist` packages.
This change adds:
* vendors-in reproducible.py script that repacks .tar.gz package in reproducible way using source-date-epoch as timestamps
* breeze release-management prepare-airflow-tarball command
* adds verification of the tarballs to PMC verification process
* adds --use-local-hatch for package building command to allow for faster / non-docker build of packages for PMC verification
* improves diagnostic output of the release and build commands
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #36744:
URL: https://github.com/apache/airflow/pull/36744#discussion_r1449919628
##########
dev/breeze/src/airflow_breeze/commands/release_candidate_command.py:
##########
@@ -320,9 +380,10 @@ def publish_release_candidate(version, previous_version, github_token):
if not github_token:
console_print("GITHUB_TOKEN is not set! Issue generation will fail.")
confirm_action("Do you want to continue?", abort=True)
- version_suffix = version[5:]
- version_branch = version[:3].replace(".", "-")
- version_without_rc = version[:5]
+
+ version_suffix = airflow_version.pre[0] + str(airflow_version.pre[1])
+ version_branch = str(airflow_version.release[0]) + "_" + str(airflow_version.release[1])
+ version_without_rc = version.base_version
Review Comment:
```suggestion
version_without_rc = airflow_version.base_version
```
##########
dev/breeze/src/airflow_breeze/commands/release_candidate_command.py:
##########
@@ -320,9 +380,10 @@ def publish_release_candidate(version, previous_version, github_token):
if not github_token:
console_print("GITHUB_TOKEN is not set! Issue generation will fail.")
confirm_action("Do you want to continue?", abort=True)
- version_suffix = version[5:]
- version_branch = version[:3].replace(".", "-")
- version_without_rc = version[:5]
+
+ version_suffix = airflow_version.pre[0] + str(airflow_version.pre[1])
+ version_branch = str(airflow_version.release[0]) + "_" + str(airflow_version.release[1])
Review Comment:
```suggestion
version_branch = str(airflow_version.release[0]) + "-" + str(airflow_version.release[1])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #36744:
URL: https://github.com/apache/airflow/pull/36744#issuecomment-1888832582
Yeah. As the next step after that, I want to turn that script into something that will be regularly run in our CI - similarly as our other release commands in Breeze - that will hopefully prevent anyu typos and makes it `all-but-guaranteed` to work always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #36744:
URL: https://github.com/apache/airflow/pull/36744#discussion_r1450376620
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
Review Comment:
Nice! Factoring in an example format is good here :)
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
```shell script
-git checkout X.Y.Zrc1
+VERSION=X.Y.Zrc1
Review Comment:
`export` not needed?
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
```shell script
-git checkout X.Y.Zrc1
+VERSION=X.Y.Zrc1
+git checkout ${VERSION}
export AIRFLOW_REPO_ROOT=$(pwd)
rm -rf dist/*
-breeze release-management prepare-airflow-package --package-format both
+breeze release-management prepare-airflow-tarball --version ${VERSION}
+breeze release-management prepare-airflow-package --package-format both --use-local-hatch
```
-That should produce `.whl` and `.tar.gz` packages in dist folder.
+Note that you need to have `hatch` installed in order to build the packages with the last command.
+If you do not have hatch, you can remove the `--use-local-hatch` flag and it will build and use
Review Comment:
```suggestion
If you do not have `hatch`, you can remove the `--use-local-hatch` flag and it will build and use
```
##########
dev/breeze/src/airflow_breeze/commands/release_management_commands.py:
##########
@@ -352,7 +378,10 @@ def prepare_airflow_packages(
# Copy all files in the dist directory in container to the host dist directory (note '/.' in SRC)
run_command(["docker", "cp", f"{container_id}:/opt/airflow/dist/.", "./dist"], check=True)
run_command(["docker", "rm", "--force", container_id], check=True)
- get_console().print("[success]Successfully prepared Airflow package!\n\n")
+ get_console().print("[success]Successfully prepared Airflow packages:")
Review Comment:
Let us have a different log from: https://github.com/apache/airflow/pull/36744/files#diff-54ea68b5e1e61ac0437df0ab7d99dc508595323b11847a957931e0d9007962d7R319 for easier tracing when it fails?
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
```shell script
-git checkout X.Y.Zrc1
+VERSION=X.Y.Zrc1
+git checkout ${VERSION}
export AIRFLOW_REPO_ROOT=$(pwd)
rm -rf dist/*
-breeze release-management prepare-airflow-package --package-format both
+breeze release-management prepare-airflow-tarball --version ${VERSION}
+breeze release-management prepare-airflow-package --package-format both --use-local-hatch
```
-That should produce `.whl` and `.tar.gz` packages in dist folder.
+Note that you need to have `hatch` installed in order to build the packages with the last command.
+If you do not have hatch, you can remove the `--use-local-hatch` flag and it will build and use
+docker image that has hatch and other necessary tools installed.
Review Comment:
```suggestion
docker image that has `hatch` and other necessary tools installed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36744:
URL: https://github.com/apache/airflow/pull/36744#discussion_r1450558579
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -585,16 +585,23 @@ Airflow supports reproducible builds, which means that the packages prepared fro
produce binary identical packages in reproducible way. You should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder:
+Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version
+you are checking):
```shell script
-git checkout X.Y.Zrc1
+VERSION=X.Y.Zrc1
Review Comment:
Not when you are using is only as parameter. I am merely using it so that I do not have to repeat it twice.
Export is only needed when you want to pass the variable version to the environment of the process you are starting as sub-process - but not when you are passing it as parameter for expansion in the same shell when your variable is defined (which is the case here as we are passing `--version ${VERSION}".
But yes, that reminds me that we have not defined it as an envariable `envvar="VERSION"`- which I just added. In this case if we exported it, we could not pass the `--version` parameter, at all (this is where `export` would be neceeassary). Still I think keepinvg `--version` as explicit flag is better in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36744:
URL: https://github.com/apache/airflow/pull/36744#discussion_r1450562760
##########
dev/breeze/src/airflow_breeze/commands/release_management_commands.py:
##########
@@ -352,7 +378,10 @@ def prepare_airflow_packages(
# Copy all files in the dist directory in container to the host dist directory (note '/.' in SRC)
run_command(["docker", "cp", f"{container_id}:/opt/airflow/dist/.", "./dist"], check=True)
run_command(["docker", "rm", "--force", container_id], check=True)
- get_console().print("[success]Successfully prepared Airflow package!\n\n")
+ get_console().print("[success]Successfully prepared Airflow packages:")
Review Comment:
Not sure I I know what you are asking for :)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update Airflow release process to include reproducible tarballs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #36744:
URL: https://github.com/apache/airflow/pull/36744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org