You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Gerrit-K (via GitHub)" <gi...@apache.org> on 2024/01/26 09:48:20 UTC
[I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Gerrit-K opened a new issue, #37026:
URL: https://github.com/apache/airflow/issues/37026
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.0
### What happened?
When accessing the "Add Dag Run" dialog (`<airflow-URL>/dagrun/add`, see screenshot below) we're facing bad UI performance (>5s response time). Upon further inspection, we realized that there were two select fields in the response that were pre-populated with _a lot_ of options: Task Instances and Creating Job. Both of them contain around 20k values in our case, which (1) heavily increases the response time and (2) makes the select box almost unusable in the browser once the page eventually renders.

### What you think should happen instead?
_No response_
### How to reproduce
Create a larger number of tasks (e.g. by running a long backfill of a multi-task DAG). Then access the mentioned view.
### Operating System
debian 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Official helm chart with official airflow image (2.8.0)
### Anything else?
Although 20k+ task instances sounds like a lot, I think this can easily happen for a production environment. However, I'm just starting to dive into Airflow, so if it is indeed wrong to have that many task instances (e.g. if they should be cleaned up) please let me know.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K closed issue #37026: Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances
URL: https://github.com/apache/airflow/issues/37026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37026:
URL: https://github.com/apache/airflow/issues/37026#issuecomment-1912604019
Yes. you can clean stuff with `airflow db clean` command (see the docs).. Not sure if the current behaviour can be easily improved. One way to do it is to replace the drop-down with search box if the number of distinct entities is reaching a threshold. Marking as good first issue, might be a good issue for someone who does web/ui stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #37026:
URL: https://github.com/apache/airflow/issues/37026#issuecomment-2626445164
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on issue #37026:
URL: https://github.com/apache/airflow/issues/37026#issuecomment-2627609285
I just tested with Airflow v2.10.4 and can't find the blue plus button anymore and accessing `/dagrun/add` manually resulted in a 404, so I assume this view has been removed. I didn't check the release notes yet, but it's also not of importance for us anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37026:
URL: https://github.com/apache/airflow/issues/37026#issuecomment-1911764394
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bad UI performance on "Add Dag Run" (/dagrun/add) with many task instances [airflow]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on issue #37026:
URL: https://github.com/apache/airflow/issues/37026#issuecomment-1914202529
> Yes. you can clean stuff with `airflow db clean` command (see the docs)
Ah, I see. Would I still be able to run backfills for dates that have been cleaned up in this case?
> One way to do it is to replace the drop-down with search box if the number of distinct entities is reaching a threshold.
Sounds like a good idea. I don't know what capabilities the Airflow UI framework has, but maybe there is already a combobox component that could be used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org