[jira] [Commented] (AIRFLOW-5523) DAGs never get cleaned up from webservers if DAG file is removed from the scheduler

["Dan Davydov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942070#comment-16942070 ] 

Dan Davydov commented on AIRFLOW-5523:
--------------------------------------

My concern with that would be that this could filter out new dags, especially ones that the user only intends to trigger manually. Maybe we can have some heuristic where if the DAG was added X days ago and still has not run we hide it, but I'm not sure that it's necessary.

I think eventually along your lines of thinking it would be nice to have a notification sent out about DAGs that have not run in a long time for clean-up/scheduler resources purposes, or maybe even going a step further and being able to automatically expiring and hiding these DAGS, although that might be better suited as a plugin or something like that.

> DAGs never get cleaned up from webservers if DAG file is removed from the scheduler
> -----------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5523
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5523
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: webserver
>    Affects Versions: 1.10.5
>            Reporter: Dan Davydov
>            Priority: Major
>         Attachments: Screen Shot 2019-09-19 at 4.39.24 PM.png
>
>
> DAGs are never removed from the DB if the DAG file used to generate the DAGs is removed. This causes the webserver to accumulate broken dag rows (see attached screenshot).
> The scheduler should delete all DAGs from the DAG table with files that no longer exist when it scans the DAGs folder for files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)