You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/12/15 10:04:02 UTC
[PR] Modify providers release process for transition to trusted publishing [airflow]
potiuk opened a new pull request, #44941:
URL: https://github.com/apache/airflow/pull/44941
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1890554834
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1197,6 +1236,38 @@ do
svn mv "${file}" "${base_file//rc[0-9]/}"
done
+# Delete the old `pypi-rc` providers
+svn rm ${SOURCE_DIR}/pypi-rc/*
+```
Review Comment:
Yeah. It's next to impossible to describe it right without actually trying it, that's why we should do it together and update the docs when it's run. It's more efficient than trying to make "dry-run" in your head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1891576969
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1233,37 +1304,6 @@ If you had this issue you will need also to make adjustments in the next step to
This is simply by removing the relevant files locally.
-## Publish the packages to PyPI
-
-By that time the packages should be in your dist folder.
-
-```shell script
-cd ${AIRFLOW_REPO_ROOT}
-git checkout <ONE_OF_THE_RC_TAGS_FOR_ONE_OF_THE_RELEASED_PROVIDERS>
-```
-
-example `git checkout providers-amazon/7.0.0rc2`
-
-Note you probably will see message `You are in 'detached HEAD' state.`
-This is expected, the RC tag is most likely behind the main branch.
-
-* Verify the artifacts that would be uploaded:
-
-```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Upload the package to PyPi:
-
-```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
Review Comment:
cool :) Sounds like a plan!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1890551881
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1197,6 +1236,38 @@ do
svn mv "${file}" "${base_file//rc[0-9]/}"
done
+# Delete the old `pypi-rc` providers
+svn rm ${SOURCE_DIR}/pypi-rc/*
Review Comment:
```suggestion
svn rm ${SOURCE_DIR}/pypi-rc/*
svn commit -m "Promote RC artifacts for Airflow Providers to released $(date "+%Y-%m-%d%n")"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1890551881
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1197,6 +1236,38 @@ do
svn mv "${file}" "${base_file//rc[0-9]/}"
done
+# Delete the old `pypi-rc` providers
+svn rm ${SOURCE_DIR}/pypi-rc/*
Review Comment:
```suggestion
svn commit -m "Promote RC artifacts for Airflow Providers to released $(date "+%Y-%m-%d%n")"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1895023839
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -483,6 +489,37 @@ twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
* Confirm that the packages are available under the links printed and look good.
+> [!NOTE]
+> The next steps until "Add tags in git" is part of the future Trusted Publishing process which we dry-run
+
+* Move and publish the `PyPI` packages to SVN
+
+```shell script
+# First clone the repo if you do not have it
+cd ..
+[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
+svn update --set-depth=infinity asf-dist/dev/airflow
+
+# Create a new folder for the release.
+mkdir -p asf-dist/dev/airflow/providers/pypi-rc
+cd asf-dist/dev/airflow/providers/pypi-rc
+
+# Remove previously released providers
+svn rm *
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* .
+
+# Add and commit
+svn add *
Review Comment:
Actually not. We moved the "SVN" packages to SVN, the "PyPI" packages are created again few lines before:
```
breeze release-management prepare-provider-packages \
--version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
```
Those will have `rc1` extension unlike SVN packages which might become the final release.
BTW. We should likely change names of those "types" of packages - previously for release candidates we commited only "SVN" packages to "SVN" and "PyPI" packages for PyPI - but now both will be committed to SVN.
So we likely need new names. Maybe:
* "no-suffix" packages
* "suffix" packages
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1890549212
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1233,37 +1304,6 @@ If you had this issue you will need also to make adjustments in the next step to
This is simply by removing the relevant files locally.
-## Publish the packages to PyPI
-
-By that time the packages should be in your dist folder.
-
-```shell script
-cd ${AIRFLOW_REPO_ROOT}
-git checkout <ONE_OF_THE_RC_TAGS_FOR_ONE_OF_THE_RELEASED_PROVIDERS>
-```
-
-example `git checkout providers-amazon/7.0.0rc2`
-
-Note you probably will see message `You are in 'detached HEAD' state.`
-This is expected, the RC tag is most likely behind the main branch.
-
-* Verify the artifacts that would be uploaded:
-
-```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Upload the package to PyPi:
-
-```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
Review Comment:
It's not removed - it's moved up. And we are changing the order because we want to also publish the `PyPI` packages in SVN. So far we have not been doing it - because we did not need them in SVN (and it was a bit problematic as well because RC `PyPI` packages never went through our SVN and were never available there nor available in the history.
With "publish" workflow we want to make sure that packages are committed to SVN (including the PyPI packages) - that's why we are moving the "prepare" step before.
And since we eventually want to replace the "publish" step from manual to workflow, it makes sense to put the "manual" step next to the "workflow" test.
I think it could be also left below, but I found it cleaner if we have "publish" steps manual/workflow next to each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1894290833
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -483,6 +489,37 @@ twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
* Confirm that the packages are available under the links printed and look good.
+> [!NOTE]
+> The next steps until "Add tags in git" is part of the future Trusted Publishing process which we dry-run
+
+* Move and publish the `PyPI` packages to SVN
+
+```shell script
+# First clone the repo if you do not have it
+cd ..
+[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
+svn update --set-depth=infinity asf-dist/dev/airflow
+
+# Create a new folder for the release.
+mkdir -p asf-dist/dev/airflow/providers/pypi-rc
+cd asf-dist/dev/airflow/providers/pypi-rc
+
+# Remove previously released providers
+svn rm *
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* .
+
+# Add and commit
+svn add *
Review Comment:
This will not work.
We already moved from dist to release the packages to PyPi. When you get to this step dist is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44941:
URL: https://github.com/apache/airflow/pull/44941#issuecomment-2705224498
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1891370098
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1233,37 +1304,6 @@ If you had this issue you will need also to make adjustments in the next step to
This is simply by removing the relevant files locally.
-## Publish the packages to PyPI
-
-By that time the packages should be in your dist folder.
-
-```shell script
-cd ${AIRFLOW_REPO_ROOT}
-git checkout <ONE_OF_THE_RC_TAGS_FOR_ONE_OF_THE_RELEASED_PROVIDERS>
-```
-
-example `git checkout providers-amazon/7.0.0rc2`
-
-Note you probably will see message `You are in 'detached HEAD' state.`
-This is expected, the RC tag is most likely behind the main branch.
-
-* Verify the artifacts that would be uploaded:
-
-```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Upload the package to PyPi:
-
-```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
Review Comment:
Let me know @eladkal if that explains it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1891393708
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1233,37 +1304,6 @@ If you had this issue you will need also to make adjustments in the next step to
This is simply by removing the relevant files locally.
-## Publish the packages to PyPI
-
-By that time the packages should be in your dist folder.
-
-```shell script
-cd ${AIRFLOW_REPO_ROOT}
-git checkout <ONE_OF_THE_RC_TAGS_FOR_ONE_OF_THE_RELEASED_PROVIDERS>
-```
-
-example `git checkout providers-amazon/7.0.0rc2`
-
-Note you probably will see message `You are in 'detached HEAD' state.`
-This is expected, the RC tag is most likely behind the main branch.
-
-* Verify the artifacts that would be uploaded:
-
-```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Upload the package to PyPi:
-
-```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
Review Comment:
I will make the release according to instructions in this PR, that way we can change it on the fly then merge the final version that actually works. It will be easier to write questions/comments on the fly when the PR is up and change it as we progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1890551881
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1197,6 +1236,38 @@ do
svn mv "${file}" "${base_file//rc[0-9]/}"
done
+# Delete the old `pypi-rc` providers
+svn rm ${SOURCE_DIR}/pypi-rc/*
Review Comment:
svn commit -m "Promote RC artifacts for Airflow Providers to released $(date "+%Y-%m-%d%n")"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #44941:
URL: https://github.com/apache/airflow/pull/44941#issuecomment-2544032202
I'll review tommorow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #44941:
URL: https://github.com/apache/airflow/pull/44941#discussion_r1886359996
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1197,6 +1236,38 @@ do
svn mv "${file}" "${base_file//rc[0-9]/}"
done
+# Delete the old `pypi-rc` providers
+svn rm ${SOURCE_DIR}/pypi-rc/*
+```
Review Comment:
rm without commit?
##########
dev/README_RELEASE_PROVIDER_PACKAGES.md:
##########
@@ -1233,37 +1304,6 @@ If you had this issue you will need also to make adjustments in the next step to
This is simply by removing the relevant files locally.
-## Publish the packages to PyPI
-
-By that time the packages should be in your dist folder.
-
-```shell script
-cd ${AIRFLOW_REPO_ROOT}
-git checkout <ONE_OF_THE_RC_TAGS_FOR_ONE_OF_THE_RELEASED_PROVIDERS>
-```
-
-example `git checkout providers-amazon/7.0.0rc2`
-
-Note you probably will see message `You are in 'detached HEAD' state.`
-This is expected, the RC tag is most likely behind the main branch.
-
-* Verify the artifacts that would be uploaded:
-
-```shell script
-twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Upload the package to PyPi:
-
-```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
-```
-
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
Review Comment:
I am not sure why this is removed.
Or at least why we are changing the order of the steps in the release. I find it confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modify providers release process for transition to trusted publishing [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #44941: Modify providers release process for transition to trusted publishing
URL: https://github.com/apache/airflow/pull/44941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org