You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "VladaZakharova (via GitHub)" <gi...@apache.org> on 2024/09/03 11:23:35 UTC
[I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
VladaZakharova opened a new issue, #41976:
URL: https://github.com/apache/airflow/issues/41976
### Description
The following DAG:
```
import datetime
import time
import airflow
from airflow.operators.python import PythonOperator
time.sleep(1000)
with airflow.DAG(
"import_timeout",
start_date=datetime.datetime(2022, 1, 1),
schedule=None) as dag:
def f():
print("Sleeping")
time.sleep(120)
for ind in range(10):
PythonOperator(
dag=dag,
task_id=f"sleep_120_{ind}",
python_callable=f,
)
```
is artificial example of DAG that will be timed out on import in DAG processor.
In logs of Airflow we can see that there is AirflowTaskTimeout for that DAG:
However, there is nothing exposed in Airflow UI that can give an idea of this.
The idea is to record import timeouts information in Airflow database and display it in Airflow UI (similar to DAG import errors).
Althought, it is not clear which model would be better to use, ImportError or DagWarning?
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #41976: DAG import timeouts recorded in Airflow Database and displayed in Airflow UI
URL: https://github.com/apache/airflow/issues/41976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326781843
Actually, as i have checked my problem is a little bit different from what you have mentioned: in the #29897 i can see that for Admin user ImportError can be visible on UI (so, correct permissions and roles fix everything). In my case, even if i am running it as Admin user i still cannot see any output on UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326309995
Yeah - since we have `ParseImportError` -> it should likely be passed same way. But maybe it already is? I know there is a problem with visibility of ImportErrors when DAG has access control (for obvious reasons) - which is tracked in https://github.com/apache/airflow/issues/29897 which might be connected. I think both are falling in the same camp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326283548
@potiuk @uranusjr @Taragolis
Hi all! This is another ticket for improving our Airflow UI :)
Need your advice before implementing this, do you have any ideas? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326531289
> Yes, looks very similar. But i think that problem cannot be resolved easily (for some reasons haha), so i can try to at least catch this kind of errors with AirflowTaskTimeout and output it on UI as ParseImportError. Right?
Not sure :). Actually answering to that question with certainty - means solving the problem :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326432524
Yes, looks very similar. But i think that problem cannot be resolved easily (for some reasons haha), so i can try to at least catch this kind of errors with AirflowTaskTimeout and output it on UI as ParseImportError. Right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2332004260
I think this is a bug and not a feature. This used to exist in Airflow <= 2.7, and provides critical info to DAG authors who get no information as to why their DAG isn't showing up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG import timeouts recorded in Airflow Database and displayed in Airflow UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #41976:
URL: https://github.com/apache/airflow/issues/41976#issuecomment-2326821753
Yeah - so likely creating a ParseImportError speciffically when there is a timeout should help :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org