You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2024/05/20 10:39:19 UTC
[I] Audit task failed-deps for later investigation [airflow]
eladkal opened a new issue, #39712:
URL: https://github.com/apache/airflow/issues/39712
### Body
Currently Airflow offers [failed-deps](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#failed-deps) to investigate why task isn't being scheduled. This is very helpful tool however it works only in real time according to the current entries in the metadb. Investigating past anomalies isn't supported.
Sometimes scheduling problems are "solved" on their own. It could be that pool is overcrowded or concurrency has been reached but eventually stress is reduced and tasks are scheduled, thus when you notice it and want to investigate why there was a delay to begin with your capabilities are limited as there could be many reasons.
**The needed solution:**
We should investigate the option to audit the `failed-deps` information or alternatively offer an easy way to export this information in real time to an external audit storage for later investigation.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Audit task failed-deps for later investigation [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #39712:
URL: https://github.com/apache/airflow/issues/39712#issuecomment-2120336770
Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched with additional checks that provide useful information.
Ref : https://github.com/apache/airflow/pull/38449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Audit task failed-deps for later investigation [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39712:
URL: https://github.com/apache/airflow/issues/39712#issuecomment-2120696473
> Recently I worked on this and the information is available as part of UI and API for tasks in scheduled or None state. Perhaps the API could be used for export and also enriched with additional checks that provide useful information.
>
> Ref : #38449
The UI paet is exposing failed-deps as is. It doesnt have the mechanisem to export/store the information.
There is also the question of export interval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org