You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/20 07:35:02 UTC
[GitHub] [airflow] antkey1 opened a new issue, #24553: Worker timout when delete dag from UI
antkey1 opened a new issue, #24553:
URL: https://github.com/apache/airflow/issues/24553
### Apache Airflow version
2.3.2 (latest released)
### What happened
After the update to 2.3.0, the ability to remove dags from the ui, if they had tags, disappeared.
With update 2.3.2, the problem should have been solved, but the delete now takes a very long time.
When I try to remove the DAG from the interface, I get a 504 Gateway Timeout error, since the request to the database takes a long time.
This is repeated with any DAG where there are many runs.
### What you think should happen instead
The DAG execution history should be deleted.
### How to reproduce
1. Run a dag and generate more than 1000 runs.
2. Remove dag via web ui.
### Operating System
Ubuntu 18.04.4 LTS (Bionic Beaver)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-jdbc==2.1.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-microsoft-mssql==2.1.3
apache-airflow-providers-mongo==2.3.3
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-oracle==2.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-samba==3.0.0
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4
apache-airflow-providers-trino==2.3.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Kubernetes v1.18.15
### Anything else
I tried to run the analysis of the SQL request to delete the dag manually and got the following result:
```
DELETE FROM dag_run WHERE dag_run.dag_id IN ('test') RETURNING dag_run.id
```
```
Delete on dag_run (cost=1440.76..32883.24 rows=27398 width=6) (actual time=2169082.358..2169985.242 rows=26750 loops=1)
-> Bitmap Heap Scan on dag_run (cost=1440.76..32883.24 rows=27398 width=6) (actual time=7.132..319.677 rows=26750 loops=1)
Recheck Cond: ((dag_id)::text = 'test'::text)
Heap Blocks: exact=6157
-> Bitmap Index Scan on dag_id_state (cost=0.00..1433.91 rows=27398 width=0) (actual time=5.963..5.963 rows=26750 loops=1)
Index Cond: ((dag_id)::text = 'test'::text)
Planning Time: 0.134 ms
Trigger for constraint task_instance_dag_run_fkey on dag_run: time=1694.339 calls=26750
Trigger for constraint task_reschedule_dr_fkey on dag_run: time=408.320 calls=26750
Trigger for constraint task_reschedule_ti_fkey on task_instance: time=2103.582 calls=53500
Trigger for constraint task_map_task_instance_fkey on task_instance: time=1802.304 calls=53500
Trigger for constraint xcom_task_instance_fkey on task_instance: time=2264561.198 calls=53500
Trigger for constraint rtif_ti_fkey on task_instance: time=5526.311 calls=53500
Trigger for constraint task_fail_ti_fkey on task_instance: time=3081.837 calls=53500
Execution Time: 4449310.647 ms
```
It runs for a very long time, it is unrealistic to increase the worker timeout to such a value.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24553: Worker timout when delete dag from UI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24553:
URL: https://github.com/apache/airflow/issues/24553#issuecomment-1160129902
This is fixed with https://github.com/apache/airflow/pull/24488 in the upcoming 2.3.3. Closing as duplicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24553: Worker timout when delete dag from UI
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24553: Worker timout when delete dag from UI
URL: https://github.com/apache/airflow/issues/24553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24553: Worker timout when delete dag from UI
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24553:
URL: https://github.com/apache/airflow/issues/24553#issuecomment-1160082381
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org