You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vghar-bh (via GitHub)" <gi...@apache.org> on 2023/07/12 04:19:37 UTC
[GitHub] [airflow] vghar-bh opened a new issue, #32540: Variables are getting duplicated on upgrading airflow
vghar-bh opened a new issue, #32540:
URL: https://github.com/apache/airflow/issues/32540
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
variables are getting duplicated on upgrading airflow to 2.5.3 from 2.4.1.
after upgrade when the airflow-sync-variables pod runs its not finding the existing variables and then adding it again.
**logs from airflow-sync-variables:**
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[[34m2023-07-09 05:59:12,443[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m
[[34m2023-07-09 05:59:14,521[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TEMP-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,534[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TEMP-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,538[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`PERSPECTIVE-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,546[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`PERSPECTIVE-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,550[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TAMR_INPUT_BUCKET_NAME` is missing, adding...[0m
[[34m2023-07-09 05:59:14,553[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TAMR_INPUT_BUCKET_NAME` was successfully added.[0m
[[34m2023-07-09 05:59:14,562[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TAMR_OUTPUT_BUCKET_NAME` is missing, adding...[0m
[[34m2023-07-09 05:59:14,573[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TAMR_OUTPUT_BUCKET_NAME` was successfully added.[0m
[[34m2023-07-09 05:59:14,577[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`RAW-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,580[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`RAW-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,585[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`EXTRACTED-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,589[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`EXTRACTED-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,592[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`CATEGORIZED-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,596[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`CATEGORIZED-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,600[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STRUCTURED-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,603[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STRUCTURED-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,608[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STANDARD-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,612[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STANDARD-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,616[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`NPD_DATE` is missing, adding...[0m
[[34m2023-07-09 05:59:14,619[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`NPD_DATE` was successfully added.[0m
[[34m2023-07-09 05:59:14,623[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-KUBERNETES-ANNOTATION` is missing, adding...[0m
[[34m2023-07-09 05:59:14,631[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-KUBERNETES-ANNOTATION` was successfully added.[0m
[[34m2023-07-09 05:59:14,635[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-DRIVER-MEMORY` is missing, adding...[0m
[[34m2023-07-09 05:59:14,638[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-DRIVER-MEMORY` was successfully added.[0m
[[34m2023-07-09 05:59:14,641[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-MEMORY` is missing, adding...[0m
[[34m2023-07-09 05:59:14,645[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-MEMORY` was successfully added.[0m
[[34m2023-07-09 05:59:14,648[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` is missing, adding...[0m
[[34m2023-07-09 05:59:14,653[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` was successfully added.[0m
[[34m2023-07-09 05:59:14,656[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-NUM-EXECUTORS` is missing, adding...[0m
[[34m2023-07-09 05:59:14,659[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-NUM-EXECUTORS` was successfully added.[0m
[[34m2023-07-09 05:59:14,662[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`DATACURATION-OUTPUT-BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,665[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`DATACURATION-OUTPUT-BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,671[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_USER` is missing, adding...[0m
[[34m2023-07-09 05:59:14,676[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_USER` was successfully added.[0m
[[34m2023-07-09 05:59:14,694[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m
[[34m2023-07-09 05:59:14,698[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m
[[34m2023-07-09 05:59:14,710[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`MYID_BASE_URL` is missing, adding...[0m
[[34m2023-07-09 05:59:14,714[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`MYID_BASE_URL` was successfully added.[0m
[[34m2023-07-09 05:59:14,727[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`DATALAKE_BUCKET` is missing, adding...[0m
[[34m2023-07-09 05:59:14,731[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`DATALAKE_BUCKET` was successfully added.[0m
[[34m2023-07-09 05:59:14,735[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`BITPULSE_WEBBITS_DATALAKE_PATH` is missing, adding...[0m
[[34m2023-07-09 05:59:14,744[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`BITPULSE_WEBBITS_DATALAKE_PATH` was successfully added.[0m
[[34m2023-07-09 05:59:14,749[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_DOCKER_IMAGE` is missing, adding...[0m
[[34m2023-07-09 05:59:14,753[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_DOCKER_IMAGE` was successfully added.[0m
[[34m2023-07-09 05:59:14,757[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_NAMESPACE` is missing, adding...[0m
[[34m2023-07-09 05:59:14,760[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_NAMESPACE` was successfully added.[0m
[[34m2023-07-09 05:59:14,765[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_MAPPING_FILE_PATH` is missing, adding...[0m
[[34m2023-07-09 05:59:14,768[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_MAPPING_FILE_PATH` was successfully added.[0m
[[34m2023-07-09 05:59:14,774[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m
[[34m2023-07-09 06:00:14,864[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m
[[34m2023-07-09 06:00:14,917[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STRUCTURED-BUCKET` is missing, adding...[0m
[[34m2023-07-09 06:00:14,921[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STRUCTURED-BUCKET` was successfully added.[0m
[[34m2023-07-09 06:00:14,952[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` is missing, adding...[0m
[[34m2023-07-09 06:00:14,958[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` was successfully added.[0m
[[34m2023-07-09 06:00:14,990[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m
[[34m2023-07-09 06:00:14,994[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m
[[34m2023-07-09 06:00:15,013[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`MYID_REDIRECT_BASE_URL` is missing, adding...[0m
[[34m2023-07-09 06:00:15,016[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`MYID_REDIRECT_BASE_URL` was successfully added.[0m
[[34m2023-07-09 06:00:15,035[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_DOCKER_IMAGE` is missing, adding...[0m
[[34m2023-07-09 06:00:15,040[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_DOCKER_IMAGE` was successfully added.[0m
[[34m2023-07-09 06:00:15,044[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_NAMESPACE` is missing, adding...[0m
[[34m2023-07-09 06:00:15,048[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_NAMESPACE` was successfully added.[0m
[[34m2023-07-09 06:00:15,058[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m
[[34m2023-07-09 06:01:15,132[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m
[[34m2023-07-09 06:01:15,256[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m
[[34m2023-07-09 06:01:15,260[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m
[[34m2023-07-09 06:01:15,323[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m
[[34m2023-07-09 06:02:15,419[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m
### What you think should happen instead
After upgrade when the airflow-sync-variables pod runs it should finding the existing variables and skip creating it again.
### How to reproduce
downgrade the airflow to 2.4.1 where its working
upgrade it to airflow 2.5.3 once we sync the airflow-sync-variables pod the variables will be duplicated in the UI and in database variable table.
### Operating System
N/A
### Versions of Apache Airflow Providers
**on airflow 2.4.1 before upgrade**
apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-apache-livy 2.2.3
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-cncf-kubernetes 4.4.0
apache-airflow-providers-common-sql 1.2.0
apache-airflow-providers-docker 3.2.0
apache-airflow-providers-elasticsearch 4.2.1
apache-airflow-providers-ftp 3.1.0
apache-airflow-providers-google 8.3.0
apache-airflow-providers-grpc 3.0.0
apache-airflow-providers-hashicorp 3.1.0
apache-airflow-providers-http 4.0.0
apache-airflow-providers-imap 3.0.0
apache-airflow-providers-microsoft-azure 4.3.0
apache-airflow-providers-mysql 3.2.1
apache-airflow-providers-odbc 3.1.2
apache-airflow-providers-postgres 5.2.2
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-sendgrid 3.0.0
apache-airflow-providers-sftp 4.1.0
apache-airflow-providers-slack 5.1.0
apache-airflow-providers-sqlite 3.2.1
apache-airflow-providers-ssh 3.2.0
**on 2.5.3 after upgrade**
apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-apache-livy 2.2.3
apache-airflow-providers-celery 3.2.0
apache-airflow-providers-cncf-kubernetes 7.0.0
apache-airflow-providers-common-sql 1.5.1
apache-airflow-providers-docker 3.7.0
apache-airflow-providers-elasticsearch 4.5.0
apache-airflow-providers-ftp 3.4.1
apache-airflow-providers-google 10.1.1
apache-airflow-providers-grpc 3.2.0
apache-airflow-providers-hashicorp 3.4.0
apache-airflow-providers-http 4.4.1
apache-airflow-providers-imap 3.2.1
apache-airflow-providers-microsoft-azure 6.1.1
apache-airflow-providers-odbc 3.3.0
apache-airflow-providers-postgres 5.5.0
apache-airflow-providers-redis 3.2.0
apache-airflow-providers-sendgrid 3.2.0
apache-airflow-providers-sftp 4.3.0
apache-airflow-providers-slack 7.3.0
apache-airflow-providers-snowflake 4.1.0
apache-airflow-providers-sqlite 3.4.1
apache-airflow-providers-ssh 3.7.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow version 2.5.3
currently working with helm charts from Airflow Helm Chart (User Community)
https://artifacthub.io/packages/helm/airflow-helm/airflow/8.7.0
using Argocd as GitOps CD tool for Kubernetes.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vghar-bh closed issue #32540: Variables are getting duplicated on upgrading airflow
Posted by "vghar-bh (via GitHub)" <gi...@apache.org>.
vghar-bh closed issue #32540: Variables are getting duplicated on upgrading airflow
URL: https://github.com/apache/airflow/issues/32540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org