You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sanromeo (via GitHub)" <gi...@apache.org> on 2024/05/22 14:56:28 UTC
[I] DAG Import Errors remain in the WebUI even after fix error [airflow]
sanromeo opened a new issue, #39762:
URL: https://github.com/apache/airflow/issues/39762
### Apache Airflow version
2.9.1
### If "Other Airflow 2 version" selected, which one?
2.9.0, 2.9.1
### What happened?
DAG Import Errors remain in the WebUI even after fix error
### What you think should happen instead?
DAG Import Errors will disappear in WebUI after fix errors
### How to reproduce
1. Create DAG with broken parameters that will cause DAG Import Errors.
2. Look at Airflow WebUI, you will see error.
3. Fix it :)
4. Error will remain the same in WebUI but should disappear after fix.
### Operating System
MacOS or Linux
### Versions of Apache Airflow Providers
-
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "sanromeo (via GitHub)" <gi...@apache.org>.
sanromeo closed issue #39762: DAG Import Errors remain in the WebUI even after fix error
URL: https://github.com/apache/airflow/issues/39762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "dttran-glo (via GitHub)" <gi...@apache.org>.
dttran-glo commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-3193633071
I met this issue in apache/airflow:3.0.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "a-poluyanov (via GitHub)" <gi...@apache.org>.
a-poluyanov commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-3219978618
> I met this issue in apache/airflow:3.0.4
I got the same error on version 3.0.5, before that I was on 3.0.2 and never had such a problem.
If you run the airflow dags reserialize --verbose command inside the pod, it serializes all dags and does not crash with an error that hangs in the web UI, and all the dags that it processed appear in the web UI. But the error does not go away. I tried to reload all the components in k8s, but the error remains in the web UI
<img width="865" height="65" alt="Image" src="https://github.com/user-attachments/assets/eedfcdc0-603d-4487-accb-df4c9cf1146a" />
<img width="426" height="244" alt="Image" src="https://github.com/user-attachments/assets/6f8f8478-2e1d-4340-86bf-48bc0510bc89" />
<img width="881" height="460" alt="Image" src="https://github.com/user-attachments/assets/26b8f915-0c5c-429d-8d0b-a1b7dfe759dc" />
And it doesn't matter to him at all that I have already updated the DAG files for the test, in which the import error is on another line and inside the pod it processes them and everything is ok. It's just that at some point the error was recorded in the database and does not disappear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "GyuhoonK (via GitHub)" <gi...@apache.org>.
GyuhoonK commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2611326321
@sanromeo
Hello, I have same issue. How do you fix this? Could you explain `specified the folder with dags as processor_subdir`?
Do you mean that you add `--subdir` parameter when run scheduler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "sanromeo (via GitHub)" <gi...@apache.org>.
sanromeo commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2127857004
[The root cause](https://github.com/apache/airflow/pull/35956) was that we have our own DagFactory and did not use the processor_subdir parameter in errors.ImportError since we have one DAG-processor, so we specified the folder with dags as processor_subdir and the problem was solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "bersena (via GitHub)" <gi...@apache.org>.
bersena commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-3583918629
I have this issue now on google cloud composer latest version with airflow 3.1.0
Fixed import errors but still shown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2125189599
I can't reproduce this.
Please provide reproduce example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "sanromeo (via GitHub)" <gi...@apache.org>.
sanromeo commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2126697330
Sorry, it's not related to all errors, only to ValidationError with pydantic for me
airflow dags list-import-errors -o json show no errors
but in Web UI there is still error appear
Seems like PydanticImportError doesn't remove from WebUI
Also link to reddit thread:
https://www.reddit.com/r/apache_airflow/comments/14oji77/airflow_262_and_pydantic_warningserrors/
Seems it happenned after upgrate pydantic to 2.+ version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2125013747
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "sanromeo (via GitHub)" <gi...@apache.org>.
sanromeo commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2629388106
@GyuhoonK sorry for late response, I mean, I added `processor_subdir` parameter in `errors.ImportError`
We build DAGs using DagFactory and when ImportError happens it took `.yml` files with for DAG configuration which not appropriate to clean ImportErrors in Airflow metadata db.
Check this [PR](https://github.com/apache/airflow/pull/35956), maybe it helps to understand better what happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-3193897549
> I met this issue in apache/airflow:3.0.4
Did you open a new issue with all details explaining it @dttran-glo -> there is little to no value in "I have the same issue" when it has no evidences and when it is posted as comment in a closed issue.
But - on the other hand - you provide useful and reproducible error scenario in new issue, it has a chance to actually help someone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "KulykDmytro (via GitHub)" <gi...@apache.org>.
KulykDmytro commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2126908610
Data remains in `import_errors` table despite of fixed DAG code
Started to appear since 2.9.0 upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-2127406314
Check on both 2.9.0 and 2.9.1 by use [sample docker compose ](filehttps://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml)
With this DAG it worked as expected
```python
from __future__ import annotations
from datetime import datetime, timezone
from airflow.decorators import task
from airflow.models.dag import DAG
raise RuntimeError("Boooom!")
START_DATE = datetime(2024, 2, 29, tzinfo=timezone.utc)
with DAG("issue_39762", schedule="@once", tags=["issue", "39762", "dag-import"], start_date=START_DATE):
@task
def do_nothing(): ...
do_nothing()
```
Gone from both DB and UI as soon as I comment out / remove `raise RuntimeError("Boooom!")`
---
Expected to see some examples and more information about your configurations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors remain in the WebUI even after fix error [airflow]
Posted by "jco-cw-de (via GitHub)" <gi...@apache.org>.
jco-cw-de commented on issue #39762:
URL: https://github.com/apache/airflow/issues/39762#issuecomment-3860783074
we just had this error while upgrading from 2.10 to 3.0.2 (DEV, VAL PROD).
on one environment the upgrade went smoth with no issues. on the second the upgrade had this error. The timestamp from the eror seems to be the time when the servers were starting after the re-deploy. so this could be some race condition?
Dags were imported and running but the error is showing and we see no way to clear it.
we copy pasted one of the "failing" dags under another ID and it imported with no problems or error.
Anyone has a solution or worksround to clear the error display?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org