You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/18 18:45:22 UTC
[GitHub] [airflow] sonalprsd opened a new issue, #27782: Expose an API which can return dagRun status in most optimized time
sonalprsd opened a new issue, #27782:
URL: https://github.com/apache/airflow/issues/27782
### Description
Airflow/MWAA does not seem to have any scalable API for returning the status of a dagRun, the APIs states-for-dag-run or list-runs are not scaling well. To fetch the dagRun status, every team seems to have some custom solution using sns_notification or updating the status to some external data store via Airflow callbacks.
The ask is to expose an API which can return dagRun status in most optimized time/by an internal query operation and not a scan.
Discussion https://github.com/apache/airflow/discussions/27765
### Use case/motivation
My use case is to fetch the Dag status of all the Active runs and update the status tables in the system. There is a poller (with a timeout of 150s configured based on our SLA). The states-for-dag-run API seems to be doing scan operation internally. As the number of DAG runs in system increases, the time to get the status of dagRun increases further. Initially, fetching the status of 100 runs took 2.5 minutes. With increase of dagRuns in the system by 50, the fetch operation to get status for 100 dagRuns is taking more than 5 minutes.
### Related issues
NA
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1320401555
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adityamalik123 commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
Adityamalik123 commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1347096847
> Hey @Adityamalik123. Did you get a chance to start working on this task? There is absolutely no rush, I was just asking in case you need help
Thanks for checking in @vincbeck. I am planning get started (and probably wrap this task up) this week, I will definitely let you know in case i get stuck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1327890591
As discussed in https://github.com/apache/airflow/discussions/27765 just exposing updated_at should be enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "shubhi-tiwari (via GitHub) via commits" <co...@airflow.apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-6654161349
Hi, I did a code reading pass and found:
updated_at column exists on both tables with onupdate=timezone.utcnow
API filter updated_at_gte/lte is wired into get_dag_runs
No database index on dag_run.updated_at
PR #65602 covers task_instance, but nothing exists for dag_run
I’d like to submit a PR adding a composite index (updated_at, dag_id) on dag_run, mirroring the approach in #65602. Also noticed a minor typo: update_at_range → updated_at_range in the route handler.
Does this scope work for the maintainers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5623938436
Thanks for your response, Vincent. I would like to be assigned this issue. Although, progress might be slow since I am a beginner, hope that's alright. Appreciate it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5620502662
Hi @potiuk and @vincbeck, I wanted to know if this issue has been resolved or gone stale. If this is still open and relevant, I would love to start working on it. Please do let me know, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adityamalik123 commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
Adityamalik123 commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1332737672
@potiuk Can i take this task up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5624004045
We decide some time ago to stop assigning issue. Please work on it and post your PR whenever it is ready :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1332779479
assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5620550124
As far as I know this is still open
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1346895695
Hey @Adityamalik123. Did you get a chance to start working on this task? There is absolutely no rush, I was just asking in case you need help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sonalprsd commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by "sonalprsd (via GitHub)" <gi...@apache.org>.
sonalprsd commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1533427998
Hi @Adityamalik123, can you share the documentation for the API you wrote as part of this task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1327890366
Marked as good first issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-3986503532
@Adityamalik123 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]
Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-6162258354
Hi, sorry for the delay — got held up but I’m back and actively working on this. I'll be back with updates soon. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org