You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2023/10/15 11:34:50 UTC
[PR] Clean local tags in tag_providers for network issues with Github [airflow]
amoghrajesh opened a new pull request, #34951:
URL: https://github.com/apache/airflow/pull/34951
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
As reported on slack, sometimes due to network connectivity issues with Github, our tag_providers.sh script creates tags locally with errors like:
```
➜ airflow git:(main) ./dev/provider_packages/tag_providers.sh
fatal: tag 'providers-airbyte/3.4.0rc1' already exists
fatal: tag 'providers-alibaba/2.6.0rc1' already exists
fatal: tag 'providers-amazon/8.8.0rc1' already exists
fatal: tag 'providers-apache-beam/5.3.0rc1' already exists
fatal: tag 'providers-apache-cassandra/3.3.0rc1' already exists
fatal: tag 'providers-apache-drill/2.5.0rc1' already exists
fatal: tag 'providers-apache-druid/3.6.0rc1' already exists
fatal: tag 'providers-apache-flink/1.2.0rc1' already exists
fatal: tag 'providers-apache-hdfs/4.2.0rc1' already exists
fatal: tag 'providers-apache-hive/6.2.0rc1' already exists
fatal: tag 'providers-apache-impala/1.2.0rc1' already exists
fatal: tag 'providers-apache-kafka/1.2.0rc1' already exists
fatal: tag 'providers-apache-kylin/3.3.0rc1' already exists
fatal: tag 'providers-apache-livy/3.6.0rc1' already exists
fatal: tag 'providers-apache-pig/4.2.0rc1' already exists
fatal: tag 'providers-apache-pinot/4.2.0rc1' already exists
fatal: tag 'providers-apache-spark/4.2.0rc1' already exists
fatal: tag 'providers-apache-sqoop/4.1.0rc1' already exists
fatal: tag 'providers-apprise/1.1.0rc1' already exists
fatal: tag 'providers-arangodb/2.3.0rc1' already exists
fatal: tag 'providers-asana/2.3.0rc1' already exists
fatal: tag 'providers-atlassian-jira/2.2.0rc1' already exists
fatal: tag 'providers-celery/3.4.0rc1' already exists
fatal: tag 'providers-cloudant/3.3.0rc1' already exists
fatal: tag 'providers-cncf-kubernetes/7.7.0rc1' already exists
fatal: tag 'providers-common-sql/1.8.0rc1' already exists
fatal: tag 'providers-daskexecutor/1.1.0rc1' already exists
fatal: tag 'providers-databricks/4.6.0rc1' already exists
fatal: tag 'providers-datadog/3.4.0rc1' already exists
fatal: tag 'providers-dbt-cloud/3.4.0rc1' already exists
fatal: tag 'providers-dingding/3.3.0rc1' already exists
fatal: tag 'providers-discord/3.4.0rc1' already exists
fatal: tag 'providers-docker/3.8.0rc1' already exists
fatal: tag 'providers-elasticsearch/5.1.0rc1' already exists
fatal: tag 'providers-exasol/4.3.0rc1' already exists
fatal: tag 'providers-facebook/3.3.0rc1' already exists
fatal: tag 'providers-ftp/3.6.0rc1' already exists
fatal: tag 'providers-github/2.4.0rc1' already exists
fatal: tag 'providers-google/10.10.0rc1' already exists
fatal: tag 'providers-grpc/3.3.0rc1' already exists
fatal: tag 'providers-hashicorp/3.5.0rc1' already exists
fatal: tag 'providers-http/4.6.0rc1' already exists
fatal: tag 'providers-imap/3.4.0rc1' already exists
fatal: tag 'providers-influxdb/2.3.0rc1' already exists
fatal: tag 'providers-jdbc/4.1.0rc1' already exists
fatal: tag 'providers-jenkins/3.4.0rc1' already exists
fatal: tag 'providers-microsoft-azure/8.0.0rc1' already exists
fatal: tag 'providers-microsoft-mssql/3.5.0rc1' already exists
fatal: tag 'providers-microsoft-psrp/2.4.0rc1' already exists
fatal: tag 'providers-microsoft-winrm/3.3.0rc1' already exists
fatal: tag 'providers-mongo/3.3.0rc1' already exists
```
in cases when the tag could not be pushed. This is irritating because now the release team has to manually delete these tags. Adding ability to do this using an env variable. The default behaviour here is that the tags will be cleaned locally.
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #34951:
URL: https://github.com/apache/airflow/pull/34951#discussion_r1359881765
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
I think the original code was correct (and simpler).
```
if git push $remote "${tags[@]}"; then
```
would still output the stdout/stderr to terminal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34951:
URL: https://github.com/apache/airflow/pull/34951#discussion_r1359908651
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
I understand the point of simpler here and I second it. Would like to run it by @eladkal once too. @eladkal wdyt? I plan to bring back the HEAD to one commit before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #34951:
URL: https://github.com/apache/airflow/pull/34951#issuecomment-1763362313
@eladkal @ephraimbuddy handling this like this as per offline discussions. Do let me know what you feel
cc @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34951:
URL: https://github.com/apache/airflow/pull/34951#issuecomment-1763366538
LGTM. @eladkal ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34951:
URL: https://github.com/apache/airflow/pull/34951#discussion_r1359865489
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
Normally the script outputs:
```
➜ airflow git:(main) ./dev/provider_packages/tag_providers.sh
Enumerating objects: 82, done.
Counting objects: 100% (82/82), done.
Delta compression using up to 10 threads
Compressing objects: 100% (82/82), done.
Writing objects: 100% (82/82), 3.33 KiB | 682.00 KiB/s, done.
Total 82 (delta 79), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (79/79), done.
To https://github.com/apache/airflow.git
* [new tag] providers-airbyte/3.4.0rc1 -> providers-airbyte/3.4.0rc1
* [new tag] providers-alibaba/2.6.0rc1 -> providers-alibaba/2.6.0rc1
* [new tag] providers-amazon/8.8.0rc1 -> providers-amazon/8.8.0rc1
* [new tag] providers-apache-beam/5.3.0rc1 -> providers-apache-beam/5.3.0rc1
* [new tag] providers-apache-cassandra/3.3.0rc1 -> providers-apache-cassandra/3.3.0rc1
* [new tag] providers-apache-drill/2.5.0rc1 -> providers-apache-drill/2.5.0rc1
* [new tag] providers-apache-druid/3.6.0rc1 -> providers-apache-druid/3.6.0rc1
* [new tag] providers-apache-flink/1.2.0rc1 -> providers-apache-flink/1.2.0rc1
* [new tag] providers-apache-hdfs/4.2.0rc1 -> providers-apache-hdfs/4.2.0rc1
* [new tag] providers-apache-hive/6.2.0rc1 -> providers-apache-hive/6.2.0rc1
* [new tag] providers-apache-impala/1.2.0rc1 -> providers-apache-impala/1.2.0rc1
* [new tag] providers-apache-kafka/1.2.0rc1 -> providers-apache-kafka/1.2.0rc1
```
If I read this right now the script will not output this but `Tags pushed successfully` ?
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
Normally the script outputs:
```
➜ airflow git:(main) ./dev/provider_packages/tag_providers.sh
Enumerating objects: 82, done.
Counting objects: 100% (82/82), done.
Delta compression using up to 10 threads
Compressing objects: 100% (82/82), done.
Writing objects: 100% (82/82), 3.33 KiB | 682.00 KiB/s, done.
Total 82 (delta 79), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (79/79), done.
To https://github.com/apache/airflow.git
* [new tag] providers-airbyte/3.4.0rc1 -> providers-airbyte/3.4.0rc1
* [new tag] providers-alibaba/2.6.0rc1 -> providers-alibaba/2.6.0rc1
* [new tag] providers-amazon/8.8.0rc1 -> providers-amazon/8.8.0rc1
* [new tag] providers-apache-beam/5.3.0rc1 -> providers-apache-beam/5.3.0rc1
* [new tag] providers-apache-cassandra/3.3.0rc1 -> providers-apache-cassandra/3.3.0rc1
* [new tag] providers-apache-drill/2.5.0rc1 -> providers-apache-drill/2.5.0rc1
* [new tag] providers-apache-druid/3.6.0rc1 -> providers-apache-druid/3.6.0rc1
* [new tag] providers-apache-flink/1.2.0rc1 -> providers-apache-flink/1.2.0rc1
* [new tag] providers-apache-hdfs/4.2.0rc1 -> providers-apache-hdfs/4.2.0rc1
* [new tag] providers-apache-hive/6.2.0rc1 -> providers-apache-hive/6.2.0rc1
* [new tag] providers-apache-impala/1.2.0rc1 -> providers-apache-impala/1.2.0rc1
* [new tag] providers-apache-kafka/1.2.0rc1 -> providers-apache-kafka/1.2.0rc1
```
If I read this right now the script will not output this but `Tags pushed successfully` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34951:
URL: https://github.com/apache/airflow/pull/34951#discussion_r1359910389
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
Sure if the output is present then lets.
I will test it with next release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #34951:
URL: https://github.com/apache/airflow/pull/34951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Clean local tags in tag_providers for network issues with Github [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #34951:
URL: https://github.com/apache/airflow/pull/34951#discussion_r1359866734
##########
dev/provider_packages/tag_providers.sh:
##########
@@ -35,6 +35,18 @@ do
{ git tag "${tag}" -m "Release $(date '+%Y-%m-%d') of providers" && tags+=("$tag") ; } || true
fi
done
+
if [[ -n "${tags:-}" && "${#tags}" -gt 0 ]]; then
- git push $remote "${tags[@]}"
+ if git push $remote "${tags[@]}"; then
+ echo "Tags pushed successfully"
Review Comment:
I see your point, the standard output was being compromised. Can you try with the latest commit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org