You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eanikindfi (via GitHub)" <gi...@apache.org> on 2024/12/27 07:51:11 UTC
[I] DAG Import Errors message is dangling in web-interface [airflow]
eanikindfi opened a new issue, #45227:
URL: https://github.com/apache/airflow/issues/45227
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.2
### What happened?
<img width="420" alt="dag_report_error" src="https://github.com/user-attachments/assets/fc99412d-7ca8-4308-88b1-cd568594846e" />
We have this message in the Airflow's web-interface no matter what. This DAG can exist or not - the error still shows.
We have found many different topics across the Internet for various Airflow versions. Some of them from 2018. So users constantly witness these dangling messages, but there are no clarification for this case.
Also we don't have an understanding what is a source for this notification message. It sits in the Airflow DB and webserver is selecting it from there or other core components are generating it for webserver somehow.
### What you think should happen instead?
Even though the error message itself is far from perfect, we expect that if DAG doesn't exist anymore, the error message should disappear.
### How to reproduce
We use gitsync option for storing DAGs:
````yaml
# DAGs Config
dags:
gitSync:
enabled: true
repo: "https://my.repo/path/name.git"
branch: "master"
rev: "HEAD"
subPath: "airflowk8s/dags"
credentialsSecret: my-secret
period: 5s
maxFailures: 10
````
And our DAG is just script.py like that, nothing complicated:
`print("D0ne!")`
So I guess steps should be:
1) Create a broken DAG in remote repository;
2) Get the error message in the Airflow's web-interface;
3) Delete a broken DAG in remote repository;
4) Wait for your gitsync;
5) The error message in the Airflow's web-interface will dangle.
But this bug appears randomly, so we don't know exact reason.
### Operating System
Kubernetes v1.31.0-eks
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "yuseok89 (via GitHub)" <gi...@apache.org>.
yuseok89 commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-4268036562
It's hard to reproduce, but from what I can see on current `main`, this area has changed a lot since then.
Related changes are in #45532, #58242, and #63617, and it looks to me like the reported behavior may already be covered there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "akashniranjan64 (via GitHub)" <gi...@apache.org>.
akashniranjan64 commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2807593573
I am working with airflow version 2.9.3 and facing the similar issue with the dangling error message of import error.
If i exec to the pod and run: airflow dags list-import-errors i see the correct state but on web the error message is not going away. This means that the meta DB is having the right state but some how the error message is stuck with the web and restarting the pod is not resolving it. Rollout deployment is also not fixing the error. Deleting the helm release and redeploying not working.
I don't find any work around to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "VergeDX (via GitHub)" <gi...@apache.org>.
VergeDX commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3396148444
Clearing the `import_error` table in the Airflow database works fine for me.
Don't worry, airflow will still refill them when rescan DAGs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "anasatzemoso (via GitHub)" <gi...@apache.org>.
anasatzemoso commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3127265417
Issue summary:
1. Create a DAG with a parsing error inside files/dags/{file_name}.
2. Wait for the bundle sync (~3–4 mins), then open the Airflow UI/reload the web interface.
3. The UI highlights a parsing error using data from the `import_errors` table.
4. Now delete the faulty DAG file without fixing the error.
5. The error still shows up in the UI even though the file no longer exists.
Why is this happening ?
The issue lies in the `_refresh_dag_bundles` method of `DagFileProcessorManager`.
This condition skips bundle refresh.
```
if (
elapsed_time_since_refresh < bundle.refresh_interval
and current_version_matches_db
and previously_seen
and bundle.name not in self._force_refresh_bundles
):
self.log.info("Not time to refresh bundle %s", bundle.name)
continue
```
So if the DAG file was deleted, but the above condition is still True, the bundle isn’t refreshed and import_errors aren’t cleared. The error disappears only after the condition eventually fails and the bundle is refreshed.
Additionally, since the frontend relies on manual reloads and has no real-time update mechanism, users keep seeing stale error messages until they refresh the page after the backend has synced.
Possible Fixes:
1. Lift the orphan cleanup logic
Move up this logic so it runs regardless of the bundle refresh skip:
```
found_files = {
DagFileInfo(rel_path=p, bundle_name=bundle.name, bundle_path=bundle.path)
for p in self._find_files_in_bundle(bundle)
}
known_files[bundle.name] = found_files
self.clear_orphaned_import_errors(
bundle_name=bundle.name,
observed_filelocs={str(x.rel_path) for x in found_files},
)
```
This ensures that when a DAG file is deleted, its corresponding `import_error` entries are also cleared.
2. Lower refresh_interval
Reduce the `dag_processor.refresh_interval` to trigger syncs more frequently.
3. Make `is_stale` column atomic:
Currently, the `is_stale` column in the dag table is overloaded - it tracks both deleted DAG files and parsing errors. We should make `is_stale` track only deletions and rely on `has_import_errors` for parse failures. This separation would make the DB state clearer and reduce ambiguity. Once done, the Get Import Errors API can also be tweaked to exclude entries related to deleted files.
4. Event-driven UI updates
Improve the frontend to automatically refresh when the backend updates bundle state instead of relying on manual reloads.
@vatsrahul1001 @uranusjr Let me know how we want to proceed so that I can go ahead and implement the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "eanikindfi (via GitHub)" <gi...@apache.org>.
eanikindfi commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2814497998
> I am working with airflow version 2.9.3 and facing the similar issue with the dangling error message of import error.
>
> If i exec to the pod and run: airflow dags list-import-errors i see the correct state but on web the error message is not going away. This means that the meta DB is having the right state but some how the error message is stuck with the web and restarting the pod is not resolving it. Rollout deployment is also not fixing the error. Deleting the helm release and redeploying not working.
>
> I don't find any work around to it.
Hi! Did you try the workaround from @potiuk mentioned earlier? You need an access to airflow DB itself for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "valentinDruzhinin (via GitHub)" <gi...@apache.org>.
valentinDruzhinin commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2661495726
@potiuk could you please assign this to me? I would like to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2596981750
Yeah. I think you'd need to fix the dag first and then delete it, which is annoying - as a workaround you can delete the import_error entry related to the DAG - marked it as a good first issue for someone to tackle (if they can reproduce it of course)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2870988803
Hi @valentinDruzhinin, just following up — how are we progressing on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "anasatzemoso (via GitHub)" <gi...@apache.org>.
anasatzemoso commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3138928968
@potiuk Just to clarify - the entries for deleted DAGs do get cleared when the bundle is refreshed. The issue happens only when the bundle refresh is skipped (due to to the if condition shared above), so it's more of a temporary inconsistency — as @uranusjr pointed out.
Since AIP-66 already aims to give finer control over refresh intervals (which would help fix this more systematically), maybe we can consider not addressing this right now unless it becomes a problem in actual usage.
Your suggestion to provide a “clear all import errors” action (maybe via UI or CLI) is interesting. It could act as a manual override for now, but I’d be slightly cautious since it may also clear real errors from active DAGs temporarily. So if we do go this route, maybe it’s worth scoping it to only orphaned errors (i.e., for files that no longer exist) ?
Let me know how you'd like me to proceed. Happy to help either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "ishtiaqSamdani007 (via GitHub)" <gi...@apache.org>.
ishtiaqSamdani007 commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2667662943
Hi @eanikindfi & @potiuk , I need clarification on the correct codebase version. The [[Artifact Hub](https://artifacthub.io/packages/helm/apache-airflow/airflow)](https://artifacthub.io/packages/helm/apache-airflow/airflow) lists the version as 2.9.3, but the description mentions 2.10.2. Should I refer to the 2.10.2 codebase for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3136348513
> Honestly I don’t feel this is too worthwhile to fix. This is just a temporary glitch. Maybe somewhere down the line we can allow each dag bundle to set its refresh interval instead—and a bundle that’s cheap to refresh, such as a local directory, can simply set it very low, even 0.
That's the plan of AIP-66 which still needs to be completed https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816356
But maybe a good (and simple) solution will be to have a possibility to "clear" all import errors - regardless where they are coming from ? For those Dags that are still there and have import errors, they will soon re-appear, but for deleted once, they will not.
I think that could be a very simple fix without real side-effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3138954023
Yeah. Fine for me to wait if this works like you described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "eanikindfi (via GitHub)" <gi...@apache.org>.
eanikindfi commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2597406008
@potiuk Thanks! A workaround helped.
In the future we will try to work like that "you'd need to fix the dag first and then delete it".
But I guess it would be helpful to validate entries in this table somehow? At least, if the DAG is no longer exist, then Airflow can delete the related entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2661514972
Assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2672484231
Not sure about the question. ArtifactHub mentiond "default" version of airflow but you can change it in the chart. So I am not sure what you are asking for. All the changes in airflow are developed in main anyway, with few exception where we cannot make a change any more and backport it, we might contribute to latest stable branch for core airflow. And we are currently in the stage where we develop Airflow 3 and the rules are slighlty different - see https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "ivica-k (via GitHub)" <gi...@apache.org>.
ivica-k commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3210780789
I've had partial success with using
```
$ airflow dags delete DAG_ID
This will drop all existing records related to the specified DAG. Proceed? (y/n) y
```
I used it to clear the import error for two DAGs. One error was never seen again, the other one stayed.
Attempting to list all DAGs with import errors was not successful
```
$ airflow dags list-import-errors
No data found
```
When listing all DAGs, the one that caused the dangling error was shown like this, and **`None`** was suspicious to me.
```
$ airflow dags list
dag_id | fileloc | owners | is_paused
=====================================+==================================+=====
MY_DAG_ID | /opt/airflow/dags/repo/dags/my_dag_id.py | airflow | None
```
I have fully solved it by:
- adding a dummy DAG with the same name and ID, and let the scheduler process it.
- removing the dummy DAG (drop the commit and force-push)
- running `airflow dags delete DAG_ID` which removed 1 record from the DB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3130300477
You’ll need to change the `found_files` structure a bit since the refresh time check is before the loop. But you also can’t simply calculate all bundle files eagerly since it’d require refreshing all the bundles first, and that could be costly.
Honestly I don’t feel this is too worthwhile to fix. This is just a temporary glitch. Maybe somewhere down the line we can allow each dag bundle to set its refresh interval instead—and a bundle that’s cheap to refresh, such as a local directory, can simply set it very low, even 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-2598491054
> But I guess it would be helpful to validate entries in this table somehow? At least, if the DAG is no longer exist, then Airflow should delete the related entry.
Sure. If you wish to contribute it - feel free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "dodiyariya6-hub (via GitHub)" <gi...@apache.org>.
dodiyariya6-hub commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3763217759
Hi maintainers, I’d like to work on this issue.
Given the discussion, I plan to implement a simple and safe solution:
add a CLI command to clear/purge all import errors from the metadata DB
so users can immediately remove stale/dangling messages.
If this approach is acceptable, I’ll open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "matschaffer-roblox (via GitHub)" <gi...@apache.org>.
matschaffer-roblox commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-3465478887
thx @VergeDX - that worked for us too. I just did a `delete from import_error;` and sure enough only the actual errors came back after what I presume is the next parser run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "matschaffer-roblox (via GitHub)" <gi...@apache.org>.
matschaffer-roblox commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-4290312216
Reproduction was pretty easy for me. Just introduce some error in a dag file (`import foo` or something), then once the error appears in the UI, delete the file from disk. I haven't tested on recent Airflow release though, so could be this is already fixed for 3+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "s23asd321-del (via GitHub)" <gi...@apache.org>.
s23asd321-del commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-4678633152
Hi maintainers, I noticed there was an earlier proposal for this direction, but I do not see a current assignee or open PR for this issue.
Based on the earlier suggestion to provide a simple way to clear all stored Dag import errors, I have a small local change that adds `airflow dags clear-import-errors` with a confirmation prompt and `--yes` support. It only deletes stored `ParseImportError` rows; errors from files that still fail to parse should reappear after the next Dag parsing run.
Does this still sound like an acceptable direction before I open a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG Import Errors message is dangling in web-interface [airflow]
Posted by "venkatamandavilli-code (via GitHub)" <gi...@apache.org>.
venkatamandavilli-code commented on issue #45227:
URL: https://github.com/apache/airflow/issues/45227#issuecomment-4724960872
Thanks for reporting this.
This seems like an important usability issue because DAG import errors are often used by operators to identify active problems in the DAG repository. If the error remains visible after the broken DAG has been removed, users may continue investigating a problem that no longer exists.
From an operations perspective, stale import errors can create confusion during deployment validation, GitSync-based DAG updates, and production support activities. It would be helpful if Airflow could either automatically clear import errors when the corresponding DAG file no longer exists, or provide clearer information about where the import error is persisted and how it can be resolved.
The GitSync scenario described here is especially relevant because DAG files can be added, updated, and removed frequently. In that case, the UI should ideally reflect the current DAG repository state and not keep showing errors for deleted DAG files.
I would be interested in helping investigate this further if maintainers think this is suitable for a first contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org