You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/11 17:34:00 UTC
[GitHub] [airflow] nogkaha opened a new issue, #24972: airflow variables list takes too long to run
nogkaha opened a new issue, #24972:
URL: https://github.com/apache/airflow/issues/24972
### Apache Airflow version
2.3.2
### What happened
Hi, I got into an issue after upgrading from 2.1.2 to 2.3.2
* first I noticed that the UI in the webserver failed to show the list of variables, instead it hanged for a while and eventually crushed with .. err.
* I run the equivalent from the CLI : `airflow variables list` and it took 4m to complete
* Querying the variables directly from the DB takes less than 1s
* Getting the value of a single variable also returns immediately
* I have 301 variables which results to a file of size 3.4M when exported.
* All the variables are encrypted and I was suspecting that the decryption takes too long. I tested it by invalidating the fernet key, and indeed this time both the UI and the CLI returns the list immediately (although without the ability to access the encrypted values)
### What you think should happen instead
Since I didn't change the amount of variables nor did I experience this issue before the update, I would expect I would expect that the list of variables would be presented in a timely manner also after the update.
### How to reproduce
As mentioned, the issue started after upgrading. In order to upgrade I basically use `apache/airflow:2.3.2-python3.9` as a base image. Later when I realised that the update require also a DB migration I run the migration command as described in the docs. It run for a while, and eventually I got an error, about not being able to release a lock from the DB. However, the webserver was working so I ignored it.
### Operating System
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-microsoft-mssql==3.0.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-samba==4.0.0
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4
```
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24972: airflow variables list takes too long to run
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24972: airflow variables list takes too long to run
URL: https://github.com/apache/airflow/issues/24972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24972: airflow variables list takes too long to run
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24972:
URL: https://github.com/apache/airflow/issues/24972#issuecomment-1180681469
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org