You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/18 18:45:22 UTC

[GitHub] [airflow] sonalprsd opened a new issue, #27782: Expose an API which can return dagRun status in most optimized time

sonalprsd opened a new issue, #27782:
URL: https://github.com/apache/airflow/issues/27782

   ### Description
   
   Airflow/MWAA does not seem to have any scalable API for returning the status of a dagRun, the APIs states-for-dag-run or list-runs are not scaling well. To fetch the dagRun status, every team seems to have some custom solution using sns_notification or updating the status to some external data store via Airflow callbacks.
   
   The ask is to expose an API which can return dagRun status in most optimized time/by an internal query operation and not a scan.
   
   Discussion https://github.com/apache/airflow/discussions/27765 
   
   ### Use case/motivation
   
   My use case is to fetch the Dag status of all the Active runs and update the status tables in the system. There is a poller (with a timeout of 150s configured based on our SLA). The states-for-dag-run API seems to be doing scan operation internally. As the number of DAG runs in system increases, the time to get the status of dagRun increases further. Initially, fetching the status of 100 runs took 2.5 minutes. With increase of dagRuns in the system by 50, the fetch operation to get status for 100 dagRuns is taking more than 5 minutes.
   
   ### Related issues
   
   NA
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1320401555

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adityamalik123 commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
Adityamalik123 commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1347096847

   > Hey @Adityamalik123. Did you get a chance to start working on this task? There is absolutely no rush, I was just asking in case you need help
   
   Thanks for checking in @vincbeck. I am planning get started (and probably wrap this task up) this week, I will definitely let you know in case i get stuck.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1327890591

   As discussed in https://github.com/apache/airflow/discussions/27765 just exposing updated_at should be enough


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "shubhi-tiwari (via GitHub) via commits" <co...@airflow.apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-6654161349

   Hi, I did a code reading pass and found:
   
   updated_at column exists on both tables with onupdate=timezone.utcnow
   API filter updated_at_gte/lte is wired into get_dag_runs
   No database index on dag_run.updated_at
   PR #65602 covers task_instance, but nothing exists for dag_run
   
   I’d like to submit a PR adding a composite index (updated_at, dag_id) on dag_run, mirroring the approach in #65602. Also noticed a minor typo: update_at_range → updated_at_range in the route handler.
   
   Does this scope work for the maintainers?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5623938436

   Thanks for your response, Vincent. I would like to be assigned this issue. Although, progress might be slow since I am a beginner, hope that's alright. Appreciate it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5620502662

   Hi @potiuk and @vincbeck, I wanted to know if this issue has been resolved or gone stale. If this is still open and relevant, I would love to start working on it. Please do let me know, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adityamalik123 commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
Adityamalik123 commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1332737672

   @potiuk Can i take this task up?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5624004045

   We decide some time ago to stop assigning issue. Please work on it and post your PR whenever it is ready :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1332779479

   assigned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-5620550124

   As far as I know this is still open


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] vincbeck commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
vincbeck commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1346895695

   Hey @Adityamalik123. Did you get a chance to start working on this task? There is absolutely no rush, I was just asking in case you need help


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] sonalprsd commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by "sonalprsd (via GitHub)" <gi...@apache.org>.
sonalprsd commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1533427998

   Hi @Adityamalik123, can you share the documentation for the API you wrote as part of this task?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27782: Expose an API which can return dagRun status in most optimized time

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-1327890366

   Marked as good first issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-3986503532

   @Adityamalik123 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
   
   This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
   
   We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Expose an API which can return dagRun status in most optimized time [airflow]

Posted by "shubhi-tiwari (via GitHub)" <gi...@apache.org>.
shubhi-tiwari commented on issue #27782:
URL: https://github.com/apache/airflow/issues/27782#issuecomment-6162258354

   Hi, sorry for the delay — got held up but I’m back and actively working on this. I'll be back with updates soon. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org